Recording intelligenceAI-generated brief · check the source for context

16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ

45:12 recording · AUTO · 1 speaker

Watch the original

Executive Summary AI
  • Shrayansh opens a deep-dive on distributed messaging queues, Kafka and RabbitMQ, pitched at high-level design interviews and the follow-up questions interviewers use to catch candidates out.
  • He gives three reasons a queue is needed: asynchronous work, where an e-commerce purchase hands notification sending to a queue instead of waiting on it; retry capability, when the send-notification application is down; and pace matching, where producers pushing 10, 20 and 30 messages per second feed a consumer that can only process 15, the same problem as city cabs reporting their car id and loca
  • In point to point a message put on the queue is consumed only once by a single consumer, while in pub sub an exchange broadcasts the same message to all bound queues so multiple consumers process it, and the business need decides which design you pick.
  • Kafka is then built up component by component: a broker is a Kafka server holding topics, a topic holds partitions, a partition holds offsets, consumers in one consumer group must read different partitions of a topic while a second group can read the same ones, a group of brokers on different machines is a cluster, and ZooKeeper lets the brokers know which topic and partition sits in which broker
  • For failure handling a buggy message does not advance the committed offset and is retried a set number of times, three in his example, then moved to a failure or dead letter queue for someone to fix and put back, while RabbitMQ does the same job with exchanges and routing keys, fan out, direct and topic types, requeueing to the back of the queue instead of offsets, and pushing to consumers rather

Brief overview

Kafka and RabbitMQ explained end to end for interviews: why queues exist and how they survive failure.

  1. A queue buys async, retry and pace matchingThose three advantages are why the e-commerce app publishes instead of calling the notification service directly.
  2. Committed offset makes a dead consumer survivableWith committed offset 3, a free consumer in the same group takes over partition zero and resumes from 4.
  3. Kafka pulls, RabbitMQ pushesKafka consumers poll for new messages while a RabbitMQ queue pushes as soon as the message arrives.

Questions this recording answers

10 questions, each answered where it is said
How is a failed Kafka message retried, and what is a dead letter queue?

If a buggy message at offset 7 fails, the committed offset stays at 6, so the consumer pulls offset 7 again and raises its retry count. You can configure three, four or five retries. Once the limit is crossed, the message goes to a failure or dead letter queue, the offset moves on, and someone can fix and re-publish it later.

What happens in Kafka when a consumer goes down?

Kafka picks another free consumer from the same consumer group. It checks the failed consumer's last committed offset, say 3, which Zookeeper keeps per group, consumer, topic and partition. The new consumer takes over partition 0 and starts reading from offset 4, continuing where the previous one left off. This is why consumer groups exist.

What happens to messages if a Kafka partition or broker goes down?

Each partition has replicas on other brokers. The main copy is the leader and the replicas are followers. All reads and writes go through the leader, and followers keep syncing every new message from it. If the leader goes down, a follower takes over as the new leader and already holds all the messages, so nothing is lost.

What is the difference between point-to-point and pub/sub messaging?

In point-to-point, a message in the queue can be consumed by only one consumer, and processed only once; if consumer 1 takes it, consumer 2 cannot. In pub/sub, an exchange broadcasts the message to all queues based on its logic, so the same message is processed by multiple consumers. You pick based on business need.

How does Kafka decide which partition a message goes to?

A message has a value, an optional key, an optional partition, and a mandatory topic. If a key such as a car id is present, Kafka hashes it and pushes to the partition the hash points to. If there is no key, it uses a specified partition. If neither is given, it assigns partitions round robin.

How does Kafka's pull model differ from RabbitMQ's push model?

Kafka is pull based: the consumer keeps polling, asking whether there is any new message. RabbitMQ is push based: the queue pushes the message to the consumer as soon as it arrives. The speaker stresses remembering this as a key difference between the two systems for interviews.

What exchange types does RabbitMQ offer?

A fanout exchange broadcasts a message to every queue bound to it. A direct exchange routes only when the message key exactly matches the routing key, for example key 1 to queue 1. A topic exchange allows wildcards, so a binding like star 1 2 3 matches India_123; wildcards are not allowed in direct exchanges.

Why use a message queue instead of calling a service directly?

It enables asynchronous work: an e-commerce app can drop a send-notification message in a queue while a notification service sends the mail, so the user does not wait and latency drops. A queue also adds retry capability: if the notification service is down, the message can be put back in the queue and retried.

How does a message queue help when producers are faster than consumers?

This is pace matching. If an e-commerce app, an inventory app and another app send 10, 20 and 30 messages per second but the notification service handles only 15, they put messages in a queue and the consumer reads at its own pace. Cab GPS updates every 10 seconds feeding a dashboard are another example.

Can two consumers in the same Kafka consumer group read one partition?

No. Within a consumer group, different partitions are read by different consumers: if consumer 1 reads partition 0, consumer 2 takes partition 1 and cannot also take partition 0. Consumers in a different consumer group, however, can still read the same topic partitions 0 and 1 independently.

Key Quote

“broker is nothing but a server kafka server”

— Shrayansh, 16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ, at 12:47▶ watch at 12:47
Key Quote

“committed offset means 3 means till 3 it is read. Only from four there are unread messages are present.”

— Shrayansh, 16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ, at 24:40▶ watch at 24:40
Key Quote

“all read and write happens in through leader”

— Shrayansh, 16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ, at 31:13▶ watch at 31:13
Key Quote

“When a queue goes down, nothing will happen to the message.”

— Shrayansh, 16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ, at 33:32▶ watch at 33:32
Key Quote

“queue which is known as failure or dead letter queue”

— Shrayansh, 16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ, at 36:33▶ watch at 36:33
Key Quote

“Kafka is pull based approach. RabbitMQ is push based approach.”

— Shrayansh, 16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ, at 37:37▶ watch at 37:37

Cite this

“16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ.” https://www.youtube.com/watch?v=oVZtzZVe9Dg. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/16-system-design-distributed-messaging-queue-design-messagin.html.

Report this page
What is wrong with this page?