Kafka and RabbitMQ explained end to end for interviews: why queues exist and how they survive failure.
If a buggy message at offset 7 fails, the committed offset stays at 6, so the consumer pulls offset 7 again and raises its retry count. You can configure three, four or five retries. Once the limit is crossed, the message goes to a failure or dead letter queue, the offset moves on, and someone can fix and re-publish it later.
Kafka picks another free consumer from the same consumer group. It checks the failed consumer's last committed offset, say 3, which Zookeeper keeps per group, consumer, topic and partition. The new consumer takes over partition 0 and starts reading from offset 4, continuing where the previous one left off. This is why consumer groups exist.
Each partition has replicas on other brokers. The main copy is the leader and the replicas are followers. All reads and writes go through the leader, and followers keep syncing every new message from it. If the leader goes down, a follower takes over as the new leader and already holds all the messages, so nothing is lost.
In point-to-point, a message in the queue can be consumed by only one consumer, and processed only once; if consumer 1 takes it, consumer 2 cannot. In pub/sub, an exchange broadcasts the message to all queues based on its logic, so the same message is processed by multiple consumers. You pick based on business need.
A message has a value, an optional key, an optional partition, and a mandatory topic. If a key such as a car id is present, Kafka hashes it and pushes to the partition the hash points to. If there is no key, it uses a specified partition. If neither is given, it assigns partitions round robin.
Kafka is pull based: the consumer keeps polling, asking whether there is any new message. RabbitMQ is push based: the queue pushes the message to the consumer as soon as it arrives. The speaker stresses remembering this as a key difference between the two systems for interviews.
A fanout exchange broadcasts a message to every queue bound to it. A direct exchange routes only when the message key exactly matches the routing key, for example key 1 to queue 1. A topic exchange allows wildcards, so a binding like star 1 2 3 matches India_123; wildcards are not allowed in direct exchanges.
It enables asynchronous work: an e-commerce app can drop a send-notification message in a queue while a notification service sends the mail, so the user does not wait and latency drops. A queue also adds retry capability: if the notification service is down, the message can be put back in the queue and retried.
This is pace matching. If an e-commerce app, an inventory app and another app send 10, 20 and 30 messages per second but the notification service handles only 15, they put messages in a queue and the consumer reads at its own pace. Cab GPS updates every 10 seconds feeding a dashboard are another example.
No. Within a consumer group, different partitions are read by different consumers: if consumer 1 reads partition 0, consumer 2 takes partition 1 and cannot also take partition 0. Consumers in a different consumer group, however, can still read the same topic partitions 0 and 1 independently.
“broker is nothing but a server kafka server”
“committed offset means 3 means till 3 it is read. Only from four there are unread messages are present.”
“all read and write happens in through leader”
“When a queue goes down, nothing will happen to the message.”
“queue which is known as failure or dead letter queue”
“Kafka is pull based approach. RabbitMQ is push based approach.”
“16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ.” https://www.youtube.com/watch?v=oVZtzZVe9Dg. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/16-system-design-distributed-messaging-queue-design-messagin.html.