Distributed Message Broker Design: Storage & Failures

This article explains how distributed message brokers handle storage and failure modes under the hood, covering the core design patterns, trade-offs, and operational realities. You’ll learn about log-based storage, replication, partitioning, and what actually happens when a broker node or disk fails.

Background: What a Distributed Message Broker Does

A message broker is middleware that lets applications send and receive messages asynchronously. Producers publish messages, consumers subscribe and process them, and the broker sits in between, decoupling the two. In a distributed setting, the broker runs as a cluster of nodes, so it must handle network partitions, node crashes, and disk failures without losing messages or blocking the entire system.

The most influential design in modern brokers is the commit log, popularized by Apache Kafka. In this model, each topic is a partitioned, ordered log. Messages are appended sequentially to a partition, and consumers read from an offset. This design contrasts with traditional queuing systems like RabbitMQ or ActiveMQ, which use per-message routing and often delete messages after consumption. The log model makes storage and replication more straightforward, which is why many newer brokers (Pulsar, Redpanda, NATS JetStream) adopt it.

Storage Architecture: Logs, Segments, and Indexes

At the core of a log-based broker is the partition log. Each partition is an ordered, immutable sequence of messages, stored on disk as a set of segments. Segments are typically fixed-size files (e.g., 1 GB in Kafka) that are written sequentially. When a segment reaches its size limit, a new one is created, and the old one can be deleted or compacted based on retention policy.

To allow fast reads by offset, each segment has an index file that maps offsets to byte positions. The broker keeps the index in memory for hot segments, and the OS page cache handles most read caching. Writes go to a single active segment, which is appended to with fsync at configurable intervals to balance durability and throughput.

Some brokers, like Pulsar, separate storage from serving: they use Apache BookKeeper as a distributed log, where each segment is replicated across bookies. This adds a layer of abstraction but also complexity. Others, like Redpanda, use a single binary that combines the broker and storage, but still follow the log-segment model.

Replication and Leader Election

To survive node failures, each partition is replicated across multiple brokers. One replica is the leader, and the others are followers. Producers write to the leader; followers pull the new messages and apply them to their own logs. Consumers also read from the leader to guarantee linearizability (in Kafka’s model).

When a leader fails, the cluster must elect a new leader from the in-sync replicas (ISR). In Kafka, the controller (a broker with special duties) handles this, using a combination of ZooKeeper or KRaft (in newer versions) for coordination. The election is quorum-based: a follower must have caught up to the last committed offset to be eligible. This ensures no message loss, but it can cause availability issues if the leader and most followers are in the same failed rack.

Pulsar uses a different approach: it separates serving and storage, so the broker that serves a partition can be different from the bookies that store it. This allows more flexible placement and faster failover, but requires two systems to be healthy.

Failure Modes and Their Impact

Distributed brokers fail in several distinct ways, each with different consequences:

• Broker crash: The process dies but the disk is intact. Recovery is fast; the broker restarts, loads its segments, and catches up on replication. No data loss if the segment was flushed.
• Disk failure: The storage medium fails, losing all data on that broker. If the broker had replicas, the cluster rebuilds them from other replicas. If it was the only replica (e.g., replication factor 1), data is permanently lost.
• Network partition: The cluster splits into two groups that cannot communicate. Both sides may elect a leader for a partition, leading to a split-brain. To prevent this, brokers use a quorum: a leader must have the majority of votes. If a partition loses the quorum, it becomes unavailable, but data consistency is preserved.
• Slow consumer: Not a broker failure, but a consumer that cannot keep up. This causes the broker to accumulate messages, potentially filling the disk. Brokers have retention limits, but if the disk fills, the broker may crash.
• Zombie leader: A leader that is still running but has lost its connection to the cluster. It may continue accepting writes, causing divergence. Modern brokers use epochs to prevent this: each new leader has a higher epoch, and old leaders reject requests with a lower epoch.

Durability and Acknowledgment Semantics

Durability is a trade-off between latency and safety. Brokers offer several acknowledgment modes:

• acks=0: Producer does not wait for any acknowledgment. Fastest, but messages can be lost.
• acks=1: Leader writes to its log and acknowledges. The message is safe if the leader does not crash before flushing.
• acks=all: Leader waits for all in-sync replicas to acknowledge. Strongest guarantee, but slower.

Additionally, the broker flushes to disk periodically (e.g., every 5000 messages or 1 second in Kafka). If the broker crashes before a flush, the messages are lost, even if they were acknowledged. To avoid this, you can set flush.messages=1, but that kills throughput. Most production systems accept a small window of loss for performance.

Choosing a Broker: Trade-offs

Selecting a broker depends on your priorities:

• Throughput and ordering: Kafka and Redpanda excel at high throughput and per-partition ordering. They are ideal for event streaming and log aggregation.
• Multi-tenancy and geo-replication: Pulsar offers better isolation and built-in geo-replication across clusters, making it suitable for global deployments.
• Routing and flexibility: RabbitMQ and ActiveMQ support complex routing (topics, headers) and per-message acknowledgments. They are better for task queues than for replayable streams.
• Operational simplicity: NATS JetStream is lightweight and easy to run, but has lower throughput and fewer features.

Also consider the ecosystem: Kafka has the largest tooling and community, but its storage is not as efficient as Redpanda’s (which uses a custom storage engine). Pulsar separates compute and storage, which can reduce cost but adds operational complexity.

Common Misconceptions and Pitfalls

A frequent mistake is assuming that acks=all guarantees zero data loss. It does not: if the leader crashes after writing to its log but before flushing to disk, and all followers have not yet replicated, the message is lost. Also, if a follower is out of sync, the leader will not wait for it, so the message is only as durable as the slowest in-sync replica.

Another pitfall is ignoring retention and compaction. By default, brokers delete old segments after a time or size limit. If you need to replay historical data, you must set retention accordingly. Log compaction keeps only the latest value for each key, which is useful for state stores but not for event sourcing.

Finally, many people think that a broker is a database. It is not: it is a log, not a query engine. You cannot index messages or run complex queries. For stateful processing, you need a stream processor (like Kafka Streams) or a separate database.

Operational Best Practices

To avoid failures:

• Always run a replication factor of at least 3, and ensure the cluster has an odd number of brokers to avoid tie votes.
• Monitor disk usage and set alerts for high watermark. Use separate disks for logs and data if possible.
• Configure min.insync.replicas to at least 2 to prevent leaders from acknowledging writes when only one replica is available.
• Use a rack-aware replica placement to survive rack failures.
• Test failover regularly by killing a broker in a staging environment.

Frequently Asked Questions

Q: Can a message be lost even with replication?
Yes, if the leader crashes before flushing to disk and the followers have not replicated that message. Also, if you set acks=1, the leader may acknowledge before the message is durable.

Q: What is the difference between a queue and a log?
A queue deletes messages after consumption; a log retains them for a configurable period. Logs allow replay and multiple consumer groups, while queues are simpler but less flexible.

Q: How does a broker handle a consumer that is too slow?
The broker keeps messages for the consumer until retention expires. If the consumer falls too far behind, it may be evicted from the consumer group, and the lag increases. Eventually, messages are deleted, and the consumer misses them.

Q: What is a split-brain, and how is it prevented?
A split-brain occurs when two nodes think they are the leader. Brokers prevent it by using a quorum: a leader must have the majority of votes. If a node loses the quorum, it steps down and becomes read-only.

Q: Is Kafka a database?
No, Kafka is a distributed log. It does not support queries or indexes. It is often used as a backbone for streaming, but for stateful queries you need a separate system.

Related

• Tab Scheduler: When to Suggest Popular Tags
• Samsung Bixby Smart Speaker: What to Know
• Best Space RPGs Like Starfield for Deeper Exploration