Premium Video

This video is available to premium subscribers only

Unlock Full Access

In this chapter, we'll look at what message brokers are, what they do internally, and how to choose the right one for your system.

Suppose your company has a dozen backend services that need to exchange messages. The order service sends events to inventory, billing, and shipping. The billing service sends events to notifications and analytics. And so on.

One option is to have each service send messages directly to the others. But then every service has to solve the same problems on its own:

Each team ends up building its own storage, retry logic, and routing, and every service needs to know the address of every service it talks to. As the number of services grows, this web of direct connections becomes hard to manage.

A message broker solves this by putting one shared piece of infrastructure in the middle.

A message broker is a server, or more commonly a cluster of servers, that receives messages from producers and delivers them to consumers.

The producer does not need to know who the consumers are, where they run, or whether they are online right now. It only needs to know how to reach the broker.

Patterns vs Brokers

In earlier chapters, we looked at message queues and pub/sub. Those are messaging patterns. A message broker is the system that implements them.

RabbitMQ, Apache Kafka, Amazon SQS, Google Cloud Pub/Sub, and NATS are all examples of message brokers, and most of them support both patterns in some form.

At a high level, every broker takes on a few core responsibilities:

How a broker handles each of these responsibilities is what makes brokers different from one another. The clearest way to see that is to compare the two most widely used ones, RabbitMQ and Kafka.

In RabbitMQ, producers do not send messages directly to queues. They send messages to an exchange.

An exchange is a routing component. It looks at each incoming message and decides which queues should receive a copy. Queues are connected to an exchange through bindings, which are rules that describe which messages a queue wants.

Each message carries a routing key, which is a short string like order.created.

Exchange Types

RabbitMQ provides several types of exchanges:

Topic Exchange Example

Suppose routing keys follow the format entity.action.region, such as order.created.us.

This gives RabbitMQ flexible routing. You can add a new consumer, create a queue with the right binding, and start receiving only the messages it needs, without changing the producer.

Kafka takes a very different approach. There are no exchanges or routing rules inside the broker.

Producers write messages to a topic, and a topic is split into partitions. Each partition is an append-only log stored on disk. New messages are added to the end, and each message gets a sequential number called an offset.

Choosing a Partition

The producer decides which partition a message goes to, typically by hashing the message key. So all messages with the same key, such as the same customer ID, land in the same partition and stay in order.

Tracking Progress

Consumers read from partitions and keep track of the offset they have reached. Kafka stores these committed offsets in an internal topic, so a consumer can resume where it left off after a restart.

Reading Doesn't Delete

The most important difference is that reading a message does not delete it. Kafka keeps messages for a configured retention period, such as seven days, or until the partition reaches a size limit.

Any number of consumer groups can read the same data, each at its own position, and any of them can move back to an earlier offset to replay messages.

These two designs reflect two different philosophies.

RabbitMQ is often described as a smart broker with simple consumers. The broker handles routing, tracks every message individually, and removes it once it is acknowledged. Consumers mostly just receive messages and acknowledge them.

Kafka is closer to a simple broker with smarter consumers. The broker focuses on storing an ordered log efficiently and serving it at high throughput. Consumers decide where to read from and manage their own progress.

Neither approach is better in general.

So far, we have talked about how messages move through a broker. The next question is what happens when part of the broker fails.

A message broker sits between many services, so losing messages at this layer can affect the entire system. Brokers protect messages in two main ways.

Persistence

The first is persistence, which means writing messages to disk rather than keeping them only in memory.

In RabbitMQ, for example, a queue must be declared durable and its messages marked as persistent for them to survive a broker restart.

Replication

Writing to disk alone isn't enough. Disks can fail, and entire machines can go down. That's why production brokers also use replication, which stores copies of messages across multiple nodes.

In Kafka, each partition has a leader and one or more follower replicas, often with three replicas in total. Producers send messages to the leader, and the followers copy the data from it.

When a producer uses the acks=all setting, Kafka acknowledges the write only after all in-sync replicas have received it. Combined with a minimum number of in-sync replicas (min.insync.replicas), such as two, this helps make sure an acknowledged message survives the failure of a single broker.

RabbitMQ provides similar protection through quorum queues, which replicate messages across multiple nodes using the Raft consensus algorithm. A write is confirmed only after a majority of the replicas have accepted it.

Wait for the Acknowledgment

A producer shouldn't assume a message has been safely stored just because it sent the request. It should wait for the broker to acknowledge it.

Otherwise, a message could still be sitting in a network buffer when a failure occurs, and the producer may never know whether it was stored. In RabbitMQ, this acknowledgment is called a publisher confirm.

Replication also lets the broker keep running when a node fails.

If the node leading a Kafka partition goes down, the cluster elects one of the in-sync followers as the new leader. Producers and consumers discover the new leader and continue working. RabbitMQ quorum queues work in a similar way, electing a new leader from the remaining nodes.

The Duplicate Problem

But failover creates a subtle problem. Suppose a producer sends a message, and the broker stores it, but the leader fails before the acknowledgment reaches the producer. From the producer's point of view, the send failed, so it retries. Now the broker has the same message twice.

Kafka handles this case with the idempotent producer. Each producer gets an ID and attaches a sequence number to its messages, so the broker can detect and discard a retried duplicate. Recent Kafka versions turn this on by default.

But duplicates can still come from other places, such as consumers that crash before committing their progress. So consumers should still be idempotent.

Another important job of a message broker is handling situations where producers send messages faster than consumers can process them. As messages pile up, the broker needs somewhere to store them.

RabbitMQ: Blocking Publishers

RabbitMQ monitors its memory usage and free disk space. If memory usage goes over a configured threshold, or free disk space drops too low, it temporarily blocks publishing connections until the pressure goes down.

This slows down producers instead of letting the broker run out of resources.

Kafka: Backlogs Are Normal

Kafka handles backlogs differently. Because messages are written to disk and kept according to a retention policy, having consumers fall behind is a normal part of how Kafka works.

However, large backlogs still use storage, and if a consumer falls too far behind, older messages may be deleted before it gets a chance to process them.

That's why consumer lag is an important metric in Kafka. It tells you how far a consumer is behind the latest messages in a partition.

Kafka also supports quotas that limit how much bandwidth individual clients can use, so one busy producer can't affect everyone else.

Push vs Pull

The delivery model also plays a role in handling backpressure:

Kafka is known for its high throughput, often handling millions of messages per second across a cluster. Understanding how it gets there is useful, because many of the same techniques apply to other storage systems.

RabbitMQ's Trade-offs

RabbitMQ makes different trade-offs. It provides flexible routing and tracks delivery and acknowledgment state for individual messages, but these features need extra work per message.

RabbitMQ generally performs best when consumers keep up with incoming messages and queues stay fairly short, rather than building up large backlogs.

Both systems can handle a lot of traffic, but they're optimized for different workloads.

You do not always have to run a broker yourself. Running a broker cluster means handling upgrades, disk capacity, replication, monitoring, and failure recovery. Managed services move most of that work to a cloud provider.

The trade-off with managed services is less control and some vendor lock-in, in exchange for much less operational work. For many teams, that trade-off is worth it.

With so many options, it's better to start with your requirements rather than picking a technology first.

Ordering

Another important consideration is message ordering. Kafka guarantees ordering within a partition, while SQS FIFO keeps ordering within a message group. However, both limit how much parallelism you can get while preserving that order.

Operational Load

Finally, consider how much operational complexity your team can handle. A managed service with fewer features is often a better choice than a powerful system that your team struggles to maintain.

You Can Use More Than One

You don't have to choose just one. Many production systems use Kafka for event streaming and analytics, alongside a simpler queue for background jobs and asynchronous tasks.

A message broker brings real benefits, but it also adds costs:

For most distributed systems with multiple services, the benefits outweigh these costs. But for two services that need a direct request and an immediate answer, a broker only adds complexity.

A message broker is the infrastructure that sits between producers and consumers and handles storage, routing, delivery, and acknowledgments for them.

To use a broker reliably, you need: