Recording intelligenceAI-generated brief · check the source for context

Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained

3:48:17 recording · AUTO · 1 speaker

Watch the original

Executive Summary AI
  • The speaker starts from an Amazon order: once a user places it, the system must save the order, cut inventory from 30 units to 29, send email and SMS, notify the warehouse, generate the invoice and update analytics.
  • Doing all of that synchronously means the user waits for every microservice, one slow service slows the whole API, and one dead service fails the order outright.
  • He then walks through seven asynchronous options with pros and cons: background threads, database polling, cron jobs, webhooks, serverless tasks, message queues and event streaming, scoring each on reliable, scalable and real time.
  • A message queue holds a task that only one consumer takes and that is deleted on consumption, while an event stream holds immutable events many consumers can read and keeps them for a retention period of roughly seven to 28 days, independent of consumption.
  • Kafka itself is a distributed log file built from brokers (machines), topics (logical grouping), partitions (the physical ordered storage, ordered within a partition but not across them), consumer groups and per-consumer-group per-partition offsets.

Brief overview

Kafka is a distributed append-only event streaming log, not a message queue, and that distinction changes everything.

  1. Synchronous order processing blocks the userThe API only responds once inventory, email, SMS and analytics all finish, so one slow or dead service ruins the order.
  2. Queues delete tasks, streams retain eventsA consumed message leaves the queue; a Kafka event stays for its retention period whether or not anyone read it.
  3. Partitions are the physical unit of scaleA topic is only a logical name; partitions hold the events on disk, keep order inside themselves and spread across brokers.

Questions this recording answers

10 questions, each answered where it is said
What is Kafka?

Kafka is a log file, but a distributed one, and most of the time it acts as event streaming, so it contains events rather than tasks. Internally it has several concepts: topic, broker, partition and consumer group, which together make it a distributed system for asynchronous processing.

Is Kafka a message queue or an event stream?

Kafka can act as both, as a message queue and as an event stream, but most of the time it acts as an event streaming platform. The speaker says that for a long period he thought Kafka was a message queue, and when he learned the complete topic he found he was thinking totally wrong.

What is the difference between a message queue and event streaming?

Three major differences. In a message queue you put the task, in an event stream you put events. A message queue has only one consumer, an event stream has multiple consumers. In a message queue the task is removed once consumed, while in an event stream it stays for the retention period, so event streaming is more durable.

Are a topic and an event the same thing?

No, they are not the same. A topic is a named stream or category, while an event is a single fact that happened. A cricket topic contains events like given out, six, boundary, wide and no ball; an orders topic contains order created and order cancelled, each with order id, user id and amount.

What is a broker and why does Kafka have several of them?

In the sports example the stadium where the match happens is the broker; in technical terms a broker is a machine or server that holds the actual data. Kafka contains multiple brokers to avoid a single point of failure, because if the only broker goes down the complete system goes down. All the brokers together are the Kafka cluster.

Why do we need partitions if we already have topics?

Data is stored inside partitions, not inside the topic. A topic is only logical partitioning of data; the partition provides the physical distribution and holds the actual events. It is like keeping small bags inside a trolley bag: when one bag crosses the weight limit you move a small bag over instead of moving clothes one by one.

Is the order of events maintained across partitions?

No. Inside one partition order is maintained, because Kafka is a distributed log file where every event is appended at the end, so for match zero the six comes first and the player getting out comes later. Across partitions order is not maintained: they act independently, and syncing them would be too much work, slow things down and cause strong coupling.

What is a consumer group and what rule does it enforce?

A consumer group is one unit, like one company such as Crick Update whose employees are the consumers. Only one consumer from a consumer group can consume from a particular partition, so employees do not duplicate work on the same match. One consumer can still consume from multiple partitions, and consumers from different groups can read the same partition.

What is an offset in Kafka?

An offset is a monotonically increasing number that uniquely identifies a record within a partition: the first event gets offset zero, then one, two, three. It is kept per consumer group per partition, which is how consumer one and consumer three, sitting in different consumer groups, read the same partition independently of each other.

What options exist for asynchronous processing besides Kafka?

Seven options are covered: background jobs using application threads or an executor, database polling with worker tasks, cron jobs or scheduled workers, webhooks, serverless or task services like AWS Lambda, Cloud Tasks and Azure Functions, message queues, and event streaming. Only message queue and event streaming come out reliable, scalable and real time.

Key Quote

“In message queue, you put the task, whereas in event stream, you put the events.”

— Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained, at 40:37▶ watch at 40:37
Key Quote

“The answer to that is broker and partition.”

— Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained, at 49:23▶ watch at 49:23
Key Quote

“One consumer can consume from multiple partitions.”

— Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained, at 1:13:30▶ watch at 1:13:30
Key Quote

“Now you cannot write data into a folder.”

— Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained, at 1:25:50▶ watch at 1:25:50

Cite this

“Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained.” https://www.youtube.com/watch?v=JFPN-GwON9U. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/kafka-in-3-hours-system-design-kafka-vs-message-queue-topic-.html.

Report this page
What is wrong with this page?