Kafka is a distributed append-only event streaming log, not a message queue, and that distinction changes everything.
Synchronous order processing blocks the userThe API only responds once inventory, email, SMS and analytics all finish, so one slow or dead service ruins the order.
Queues delete tasks, streams retain eventsA consumed message leaves the queue; a Kafka event stays for its retention period whether or not anyone read it.
Partitions are the physical unit of scaleA topic is only a logical name; partitions hold the events on disk, keep order inside themselves and spread across brokers.
Executive SummaryAI
The speaker starts from an Amazon order: once a user places it, the system must save the order, cut inventory from 30 units to 29, send email and SMS, notify the warehouse, generate the invoice and update analytics.
Doing all of that synchronously means the user waits for every microservice, one slow service slows the whole API, and one dead service fails the order outright.
He then walks through seven asynchronous options with pros and cons: background threads, database polling, cron jobs, webhooks, serverless tasks, message queues and event streaming, scoring each on reliable, scalable and real time.
A message queue holds a task that only one consumer takes and that is deleted on consumption, while an event stream holds immutable events many consumers can read and keeps them for a retention period of roughly seven to 28 days, independent of consumption.
Kafka itself is a distributed log file built from brokers (machines), topics (logical grouping), partitions (the physical ordered storage, ordered within a partition but not across them), consumer groups and per-consumer-group per-partition offsets.
Key Quote
“In message queue, you put the task, whereas in event stream, you put the events.”
— Speaker
Key Quote
“The answer to that is broker and partition.”
— Speaker
Key Quote
“One consumer can consume from multiple partitions.”