When a producer sends an order event to Kafka, several components work together to make it available to consumers. The producer finds the right broker, the broker stores the record, and other brokers keep copies. Behind the scenes, controllers track the cluster’s structure and which brokers lead each partition.

In this chapter, we’ll see how those pieces fit together. We’ll follow a record in an orders.placed topic through a KRaft-based cluster, then look at how Kafka responds when a broker, controller, or consumer fails.

Producers and consumers are part of your applications. They use Kafka client libraries to send and read records. Inside the cluster, brokers store and serve those records, while controllers manage cluster metadata: information about the cluster’s structure and current state.

The main roles are:

KRaft uses the Raft consensus protocol to keep copies of cluster metadata consistent. The controllers form a metadata quorum, which uses majority agreement to make changes. One controller is active. The others copy the metadata and can take over if it fails.

The diagram shows how clients and controllers interact with brokers. The broker box represents the cluster’s brokers as a group; clients connect to the individual brokers they need.

Producers and consumers send and read records through brokers. Controllers decide which brokers lead partitions, but order events do not pass through the active controller.

A Kafka server can be a broker, a controller, or both. Running both roles in one process is useful for local practice. Running them separately lets you manage their resources and maintenance independently. Our example uses three brokers and three separate controllers to make each role easy to follow. It is an example layout, not a guide to sizing a production cluster.

A topic is a named stream that applications write to and read from. It contains partitions, and Kafka stores each partition as an ordered log. Every record has an offset that identifies its position within that partition.

A replica is a copy of a partition’s log. One replica is the leader, and the others are followers. Producers send records to the leader. Followers fetch those records and add them to their own logs.

Suppose orders.placed has two partitions, 0 and 1, and a replication factor of 3. Each partition has three copies, including the leader. That gives us six replicas in total: three copies of each of the two partition logs.

Here, each broker holds one copy of each partition. The arrows show records flowing from leaders to followers. The followers request those records by fetching from the leader.

Broker 1 leads partition 0 and follows partition 1. Broker 2 does the reverse. Leadership is per partition, so there is no single broker that leads the whole cluster. Broker 3 keeps both of its follower replicas up to date.

Partitions let Kafka spread a topic’s data and reading work across the cluster. Replicas keep extra copies for recovery. Adding replicas does not give a regular consumer group more partitions to read in parallel.

Brokers store each replica’s log in files. Kafka’s retention and cleanup policies control which records remain available. Reading or processing a record does not remove it from the log.

An application usually starts with a list of broker addresses called bootstrap servers. The client uses them to make its first connection and discover the cluster. Later requests may go to other brokers.

Suppose the order producer first connects to Broker 2. It asks for metadata and learns that Broker 1 leads partition 0 of orders.placed. If it chooses partition 0, it sends the records directly to Broker 1. Broker 2 does not forward them.

The diagram shows this first contact and the direct write that follows it.

Metadata tells clients about topic partitions, their leaders, and broker addresses. Clients keep a cached copy and refresh it when needed, such as when a partition gets a new leader.

This explains a common connection problem: a client can reach a bootstrap server but cannot reach the broker address the metadata response provides. Every broker must advertise an address that the intended clients can use.

The KRaft metadata log stores information such as which broker leads orders.placed partition 0. The partition’s own log stores the order records. Kafka needs to replicate both: metadata across controllers and topic records across brokers.

Let’s follow an OrderPlaced event with the event ID evt-7001 for order ord-1042. The producer uses ord-1042 as the key and puts the event’s fields in the value.

For this example, the producer selects partition 0, which Broker 1 leads, and the record receives offset 42. A consumer in the order-reporting group receives this partition assignment. It reads from the leader, which is the usual setup. You can also configure Kafka to let consumers fetch from followers.

The diagram shows the record’s path. In a running cluster, replication and consumer reads happen continuously, with many requests in progress at once.

Here’s what happens at each step:

Suppose the producer uses acks=all and all three replicas stay in sync. In-sync replicas are copies that Kafka considers sufficiently caught up with the leader. Before acknowledging a successful write, the leader waits until all three have the record. The min.insync.replicas setting controls the minimum number of in-sync replicas required to accept writes with acks=all.

These settings help protect records from loss, but the protection depends on which replicas remain available and what fails. A successful write also says nothing about the reporting database. The acknowledgment can reach the producer before the consumer has even read the record.

For the consumer, reading the record, updating the database, and saving progress are separate steps. If it crashes after the update but before committing, it may process the record again. Committing an offset does not make the database update and saved Kafka progress a single atomic operation.

Consumer groups let an application share the work of reading records. With two partitions and two consumers in order-reporting, one consumer can read partition 0 while the other reads partition 1. A separate group, order-notifications, can read both partitions independently to send notifications.

One broker acts as the group’s coordinator. It handles group coordination and manages committed offsets, which Kafka stores in its internal __consumer_offsets topic. This is a broker’s job, separate from the KRaft controller’s work.

Our reporting consumer reads partition 0 from Broker 1 and sends group-related requests to its coordinator. The coordinator might be Broker 1 or another broker. Records go directly from the broker serving the partition to the consumer, without passing through a separate coordinator.

When consumers join, leave, or fail, Kafka may change the partition assignments. This is a rebalance, and it lets another group member take over a partition. The details depend on the consumer group protocol.

Once assignments settle, each partition has one assigned reader within a regular consumer group. That does not guarantee a business action happens only once. A consumer may update a database and then fail before saving its progress, leaving its replacement to read the same record again. The application needs to handle that repeated work safely.

Suppose Broker 1 fails after the record at offset 42 has reached all the in-sync replicas in our example. Partition 0 loses its leader. Broker 2 continues to lead partition 1.

If the controller quorum is available and a surviving replica is eligible to lead, the active controller can choose a new leader for partition 0. Suppose it chooses Broker 2. Producers and consumers refresh their metadata and connect to Broker 2 for that partition. During the switch, requests may pause or fail, so clients need to handle retries.

A producer can also miss an acknowledgment for a write that succeeded. It then has to decide whether to retry without knowing the outcome. Producer retry settings and application duplicate handling matter here; having replica copies alone does not prevent duplicate events.

If the active controller fails, the remaining controllers can elect a replacement as long as a majority is available. For three voting controllers, that means at least two. The replacement takes over metadata management. It does not take over leadership of every topic partition.

If the controllers lose their majority, Kafka cannot make metadata changes that need quorum agreement, including partition leader changes. Some requests to existing leaders may still work temporarily, but the cluster cannot operate normally. Kafka needs both an available controller quorum and available partition replicas.

A consumer failure leaves the stored records and partition leaders unchanged. The group can assign its partitions to another consumer, which can resume from the committed offsets if the records are still available. Broker recovery restores access to partition data; consumer recovery gets the application’s processing moving again.

Kafka clients use metadata to find the brokers they need. Brokers store partition logs: leaders accept writes, and followers keep copies. KRaft controllers track the cluster’s structure and manage leadership changes.

Consumer groups share reading work and save progress through a broker coordinator. Storing a record, copying it, acknowledging the write, processing it, and committing an offset are separate steps. Understanding those steps helps you see what Kafka has confirmed and what your application still needs to handle.