Recording intelligenceAI-generated brief · check the source for context

16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ

45:12 recording · AUTO · 1 speaker

Watch the original

Brief overview

Kafka and RabbitMQ explained end to end for interviews: why queues exist and how they survive failure.

  1. A queue buys async, retry and pace matchingThose three advantages are why the e-commerce app publishes instead of calling the notification service directly.
  2. Committed offset makes a dead consumer survivableWith committed offset 3, a free consumer in the same group takes over partition zero and resumes from 4.
  3. Kafka pulls, RabbitMQ pushesKafka consumers poll for new messages while a RabbitMQ queue pushes as soon as the message arrives.
Executive Summary AI
  • Shrayansh opens a deep-dive on distributed messaging queues, Kafka and RabbitMQ, pitched at high-level design interviews and the follow-up questions interviewers use to catch candidates out.
  • He gives three reasons a queue is needed: asynchronous work, where an e-commerce purchase hands notification sending to a queue instead of waiting on it; retry capability, when the send-notification application is down; and pace matching, where producers pushing 10, 20 and 30 messages per second feed a consumer that can only process 15, the same problem as city cabs reporting their car id and loca
  • In point to point a message put on the queue is consumed only once by a single consumer, while in pub sub an exchange broadcasts the same message to all bound queues so multiple consumers process it, and the business need decides which design you pick.
  • Kafka is then built up component by component: a broker is a Kafka server holding topics, a topic holds partitions, a partition holds offsets, consumers in one consumer group must read different partitions of a topic while a second group can read the same ones, a group of brokers on different machines is a cluster, and ZooKeeper lets the brokers know which topic and partition sits in which broker
  • For failure handling a buggy message does not advance the committed offset and is retried a set number of times, three in his example, then moved to a failure or dead letter queue for someone to fix and put back, while RabbitMQ does the same job with exchanges and routing keys, fan out, direct and topic types, requeueing to the back of the queue instead of offsets, and pushing to consumers rather
Key Quote
“broker is nothing but a server kafka server”
— Shrayansh
Key Quote
“committed offset means 3 means till 3 it is read. Only from four there are unread messages are present.”
— Shrayansh
Key Quote
“all read and write happens in through leader”
— Shrayansh