How Kafka Works - and Why You Probably Don’t Need It

Every distributed systems talk eventually mentions Kafka. It shows up in system design interviews, architecture diagrams, and job descriptions. Teams reach for it when they need to “handle scale” or “decouple services”, often before they have understood what Kafka actually does or what it costs to operate.

This is what Kafka is, how it works, where it genuinely shines, and where it is the wrong tool.

What Kafka Actually Is

Kafka is a distributed log. That is the clearest way to describe it.

Producers write messages to Kafka. Consumers read those messages. The key difference from a regular message queue is that Kafka retains messages after consumers read them. A traditional queue deletes a message once it is consumed. Kafka keeps it, stored on disk, for as long as you configure it to. Multiple consumers can read the same message independently.

This single property - durability and replayability - is what makes Kafka genuinely different from tools like RabbitMQ or SQS.

Topics and Partitions

In Kafka, messages are organized into topics. A topic is a category, something like user-signups or order-events. Producers write to a topic. Consumers read from a topic.

Here is where it gets interesting. Each topic is split into partitions. A partition is an ordered, append-only log. Messages within a single partition are always in order. Messages across partitions have no guaranteed ordering.

Why split into partitions? Parallelism. If you have one partition, only one consumer can read from it at a time. If you have ten partitions, ten consumers can read in parallel, each handling their own slice of the data. Partitions are how Kafka scales.

When a producer sends a message, Kafka decides which partition it goes to. If the message has a key, Kafka hashes that key to pick a consistent partition. This ensures all messages with the same key always land in the same partition to guarantee ordering, which is related to the concepts behind consistent hashing. If there is no key, Kafka distributes messages round-robin across partitions.

Offsets

Every message in a partition gets an offset - a sequential integer starting at 0. Think of it as the position of that message in the log.

Partition 0: [0] [1] [2] [3] [4] [5] ...
Partition 1: [0] [1] [2] [3] ...

Offsets are how consumers track their position. A consumer reads a batch of messages and then commits its offset, essentially saying “I have processed everything up to position 47 in partition 0.” If that consumer crashes and restarts, it picks up from offset 48. Nothing is lost.

Kafka stores committed offsets in an internal topic called __consumer_offsets. This is how Kafka knows where each consumer left off, without the consumer needing to maintain that state itself.

Consumer Groups

A consumer group is a set of consumers that cooperate to read a topic. Kafka assigns each partition to exactly one consumer within a group. No two consumers in the same group read from the same partition simultaneously.

This is what enables horizontal scaling. Add more consumers to a group and Kafka redistributes partitions among them. One consumer might handle partitions 0 and 1, another handles 2 and 3.

There is a ceiling though. You cannot have more active consumers in a group than you have partitions. If you have five partitions and add a sixth consumer, that sixth consumer sits idle. This is why choosing your partition count upfront matters - you cannot easily reduce partitions later.

Different consumer groups are completely independent. A group called analytics-service and a group called billing-service can both read the same topic and each maintains its own offset. This is the replayability property in practice - each service sees every message without interfering with the other.

What Happens When a Consumer Goes Down

When a consumer in a group stops sending heartbeats - either because it crashed or because processing took too long - the group coordinator triggers a rebalance. Kafka reassigns that consumer’s partitions to the remaining consumers.

Rebalancing is Kafka’s Achilles heel for teams new to it. During a rebalance, all consumers in the group pause processing. In clusters with many consumers or frequent deployments, this can cause noticeable latency spikes. There is incremental rebalancing in newer Kafka versions to reduce the impact, but it is still something you have to design around.

Where Kafka Is the Right Choice

Kafka earns its complexity in specific situations.

Event sourcing at scale. If you are storing every state change as an event and need to replay them - to rebuild read models, audit, or recover from bugs - Kafka’s retention makes this natural. The log is the source of truth.

Fan-out to multiple consumers. One payment event needs to update accounting, trigger notifications, sync to a data warehouse, and feed a fraud detection model. With a regular queue, you are either duplicating messages or building a dispatcher. With Kafka, each service is its own consumer group reading the same topic.

High-throughput pipelines. Kafka was built at LinkedIn to handle billions of events per day. If you are processing telemetry, clickstreams, or IoT data at genuine scale, Kafka handles it well.

Guaranteed ordering per key. All order events for customer ID 1234 always land in the same partition, processed in sequence. This matters for inventory updates, financial transactions, and anything where out-of-order processing causes incorrect state.

Where Kafka Is Not the Right Answer

If your team is using Kafka for a microservices setup that handles a few thousand requests per day, you have added significant operational complexity without any corresponding benefit.

Kafka requires running brokers, managing ZooKeeper or KRaft, monitoring consumer lag, tuning partition counts, and handling rebalancing issues. A small engineering team spending time on Kafka operations is a team not spending time on product.

Use a managed queue (SQS, RabbitMQ, Cloud Pub/Sub) if:

• You have one consumer per message (classic task queue behavior)
• Your scale is under a few thousand messages per second
• You need retry and dead-letter queue behavior
• You do not need to replay messages

Use Kafka if:

• Multiple independent consumers need the same events
• You need replay capability (audit trails, event sourcing)
• You are handling genuinely high throughput (hundreds of thousands of events per second)
• You have the operational capacity to run and monitor it

The Part That Trips People Up

Kafka does not delete messages when consumed. This is a feature, but it means you are responsible for configuring retention. Set retention.ms too low and you lose the replay capability that justified using Kafka in the first place. Set it too high and your disk usage becomes a real cost. When containerizing Kafka, make sure to map these storage directories to persistent host volumes using tools like Docker to avoid data loss on container restart.

The other thing teams underestimate is consumer lag monitoring. Consumer lag is the difference between the latest offset in a partition and the committed offset of your consumer. When lag grows, your consumers are falling behind. Without lag monitoring, you often find out after the fact when downstream systems are hours behind real-time.

The Real Question

Before adding Kafka, answer this: what happens if you just use your existing database plus a job queue?

For most teams, that is the right answer for longer than they expect. Kafka is excellent infrastructure, but it is infrastructure - not a solution. The problem you are solving still needs to be solved regardless of the message broker you pick.

When you do reach the point where a single consumer per message is a bottleneck, where you genuinely need multiple independent consumers reading the same events, where replay is a real requirement - that is when Kafka starts paying for itself.