Kafka is a distributed append-only event streaming log, not a message queue, and that distinction changes everything.
Kafka is a log file, but a distributed one, and most of the time it acts as event streaming, so it contains events rather than tasks. Internally it has several concepts: topic, broker, partition and consumer group, which together make it a distributed system for asynchronous processing.
Kafka can act as both, as a message queue and as an event stream, but most of the time it acts as an event streaming platform. The speaker says that for a long period he thought Kafka was a message queue, and when he learned the complete topic he found he was thinking totally wrong.
Three major differences. In a message queue you put the task, in an event stream you put events. A message queue has only one consumer, an event stream has multiple consumers. In a message queue the task is removed once consumed, while in an event stream it stays for the retention period, so event streaming is more durable.
No, they are not the same. A topic is a named stream or category, while an event is a single fact that happened. A cricket topic contains events like given out, six, boundary, wide and no ball; an orders topic contains order created and order cancelled, each with order id, user id and amount.
In the sports example the stadium where the match happens is the broker; in technical terms a broker is a machine or server that holds the actual data. Kafka contains multiple brokers to avoid a single point of failure, because if the only broker goes down the complete system goes down. All the brokers together are the Kafka cluster.
Data is stored inside partitions, not inside the topic. A topic is only logical partitioning of data; the partition provides the physical distribution and holds the actual events. It is like keeping small bags inside a trolley bag: when one bag crosses the weight limit you move a small bag over instead of moving clothes one by one.
No. Inside one partition order is maintained, because Kafka is a distributed log file where every event is appended at the end, so for match zero the six comes first and the player getting out comes later. Across partitions order is not maintained: they act independently, and syncing them would be too much work, slow things down and cause strong coupling.
A consumer group is one unit, like one company such as Crick Update whose employees are the consumers. Only one consumer from a consumer group can consume from a particular partition, so employees do not duplicate work on the same match. One consumer can still consume from multiple partitions, and consumers from different groups can read the same partition.
An offset is a monotonically increasing number that uniquely identifies a record within a partition: the first event gets offset zero, then one, two, three. It is kept per consumer group per partition, which is how consumer one and consumer three, sitting in different consumer groups, read the same partition independently of each other.
Seven options are covered: background jobs using application threads or an executor, database polling with worker tasks, cron jobs or scheduled workers, webhooks, serverless or task services like AWS Lambda, Cloud Tasks and Azure Functions, message queues, and event streaming. Only message queue and event streaming come out reliable, scalable and real time.
“In message queue, you put the task, whereas in event stream, you put the events.”
“The answer to that is broker and partition.”
“One consumer can consume from multiple partitions.”
“Now you cannot write data into a folder.”
“Kafka in 3 Hours | System Design | Kafka vs Message Queue | Topic, Partition, Consumer Explained.” https://www.youtube.com/watch?v=JFPN-GwON9U. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/kafka-in-3-hours-system-design-kafka-vs-message-queue-topic-.html.