Keeping several copies of a partition raises a coordination problem: which copy decides the order of new records, and how do the others stay consistent with it? Kafka gives one replica the leader role and makes the remaining replicas followers.

These roles govern more than where a producer connects. They determine how records receive offsets, how replication advances, and what happens when a different broker takes over. We’ll follow those responsibilities through one partition in a KRaft-based Apache Kafka 4.3 cluster, using non-transactional records and local broker storage.

A partition leader is the replica responsible for accepting producer writes and establishing their order in the partition log. A follower copies that log from the leader, preserving the assigned offsets and record order.

Suppose orders.placed, partition 0, has replicas on Brokers 1, 2, and 3. Broker 1 is its current leader. Producers send writes for this partition to Broker 1, even though the other brokers also store copies.

If all three replicas independently accepted writes, two brokers could assign different records to the same next offset. They would then need a way to resolve conflicting histories. Kafka’s single-leader model gives the partition one place to establish the order that followers reproduce.

Leadership belongs to a partition, not to an entire topic or broker. Broker 1 can lead partition 0 while following partition 1, whose leader is Broker 2.

The diagram shows that relationship for two partitions. Each arrow represents replication data the leader returns in response to follower fetches. The diagram leaves out Broker 3’s copies to keep the role comparison small.

Both brokers are doing leader and follower work at once. Calling a machine “the leader broker” is therefore incomplete unless the partition is clear from the context.

The KRaft controller has a separate responsibility. It manages partition leadership in cluster metadata. Being the active controller does not make a server the leader of every topic partition, and ordinary topic writes do not pass through the controller.

Before sending records, a producer obtains metadata identifying the partition’s current leader. It connects directly to that broker and sends a Produce request for the selected partition.

For our example, an order service sends a batch of three new events to orders.placed-0. The first has key ord-1042, a value describing an OrderPlaced event with event ID evt-7001, and a source header containing order-service. The other events concern orders ord-1043 and ord-1044.

Assume the producer has routed all three records to partition 0. The leader receives their serialized bytes, validates the request and record batch, and appends the new records at offsets 42, 43, and 44.

The followers will copy those same offsets. They do not hash the keys again, choose different partitions, or assign new positions for the copied records.

Log Order and Application Order

The leader establishes an append order for the partition. If two independent producers submit events concurrently, Kafka does not infer which business event should logically come first from their timestamps or order IDs.

For example, two services might report changes to ord-1042 at almost the same time. The partition log provides a definite order once the leader appends the records. Whether that order represents valid business transitions remains an application concern.

This guarantee is also local to the partition. Partition 1 has its own leader and offset sequence; its offset 42 has no ordering relationship with offset 42 in partition 0.

Producer Responses

The producer’s acknowledgment setting determines what the leader must confirm. With acks=1, a successful response confirms the leader’s local append without waiting for followers. With acks=all, successful completion waits for the required replication condition.

For the walkthrough, assume all three replicas remain in the in-sync replica set, or ISR, and the producer uses acks=all. The ISR contains replicas Kafka considers sufficiently caught up, including the leader. Under these assumptions, the leader waits for both followers to copy the batch before acknowledging success.

A local append and replication to another broker are separate events. A successful acknowledgment also does not imply that any consumer has processed the order, or that every replica forced a physical disk flush for that batch.

Followers actively request data from the leader. Each follower tracks its local log end offset, the position immediately after the last record in its log. If it has records through offset 41, its log end offset is 42.

After Broker 1 appends our batch, its log end offset becomes 45. A follower still at 42 asks the leader for data starting there. The response carries records 42 through 44, which the follower appends locally.

The follower’s next fetch starts at 45. That request also tells the leader that this follower has advanced past the batch. Kafka learns replication progress through the follower fetch protocol; it does not need a separate acknowledgment from the follower for every individual record.

The diagram follows Broker 2 through this cycle. Broker 3 performs the same work independently.

There is a small but important delay between a follower storing a batch and the leader learning that it has done so. Network round trips and follower storage work therefore affect how quickly a replicated write can complete.

Followers copy batches continuously as new records arrive. Their work uses network bandwidth, disk throughput, and CPU even when they serve no application reads. A follower that cannot keep up may fall behind because of a slow disk, network congestion, or competing work on its broker.

Replication Lag and Consumer Lag

Replication lag measures how far a replica trails the leader. Consumer lag describes how far an application’s reading or saved progress trails the partition, depending on which metric you use.

A reporting application can be hours behind while every replica is caught up. Conversely, the reporting application may be processing quickly while one follower struggles to copy new records. Those situations involve different components and require different investigation.

A follower’s replication position is broker state. It does not join an application consumer group or commit offsets to track its copying work.

The leader’s log end tells us what it has appended locally. It does not, by itself, tell us what ordinary consumers can read.

Kafka also tracks a high watermark, an offset boundary below which records are replication-committed. Ordinary consumers read within that boundary, with an additional restriction when transaction isolation applies.

Keep the same batch and assume the ISR stays fixed at Brokers 1, 2, and 3 throughout the following timeline. Initially, all replicas contain records through offset 41, and the leader’s high watermark is 42.

With the high watermark at 42, records 42–44 remain beyond the readable boundary. Once it reaches 45, all three records are eligible for ordinary consumer reads in this non-transactional example.

The high watermark is an exclusive boundary: 45 makes records through 44 eligible. It is not the offset of the last readable record.

The fixed ISR assumption matters. Kafka manages which replicas remain in sync, so the general rule is not “wait for every assigned replica forever.” Assignment, replication progress, and ISR membership describe different states.

Two Meanings of Committed

A replication-committed record has crossed Kafka’s replication boundary. A consumer’s committed offset records where that consumer group should resume processing.

The high watermark can reach 45 while a reporting consumer still has a committed offset of 30. Kafka has made the newer data available; the application has saved progress only through the earlier records.

Even a consumer using read_uncommitted does not bypass the replication high watermark. That setting concerns transaction visibility. A read_committed consumer also waits for transaction outcomes, which can hold its readable boundary behind the high watermark.

Reads from Followers

By default, consumers fetch from partition leaders. Kafka can also select a follower for consumer reads when you configure the broker and client for follower fetching, often to keep traffic within a nearby rack or availability zone.

That follower still copies the leader’s log and respects its applicable read boundary. It does not become another writer, and its view of newly readable data can trail the leader’s while replication information propagates.

This makes follower fetching a read-placement option. It does not create additional partitions or remove the leader’s responsibility for producer writes.

Leader and follower are roles that can change during the lifetime of a replica. Broker failure, maintenance, or a leadership-balancing operation can cause Kafka to designate a different replica as leader.

Suppose Broker 1 fails after our batch is replication-committed and both followers have records through offset 44. Assume the controller quorum is available, Broker 2 is eligible, and the controller selects it to lead partition 0.

Broker 2 already has the partition log. It takes over the leader responsibilities, and Broker 3 begins fetching from it. Once producers learn the new leader, Broker 2 can append new records beginning at offset 45, assuming no other writes have occurred.

The diagram shows the role changes for this partition only. The final state assumes Broker 1 returns successfully and catches up as a follower.

This changes where requests go while preserving the topic, partition number, and committed log history under the stated recovery conditions. Consumers do not need a new partition assignment merely because the leader moved. They may need to reconnect and refresh metadata, and requests can fail or pause during the transition.

A returning Broker 1 does not regain authority simply because it used to be leader. It must follow the cluster’s current leadership state. A later leadership operation may move the role again.

Leader Epochs and Log Reconciliation

Distributed systems cannot assume every participant learns about a role change at the same instant. Kafka uses a leader epoch, a version number for a partition’s leadership, to distinguish successive leadership terms. Kafka can reject requests carrying outdated epochs, helping brokers and clients recognize stale state.

Epoch information also helps replicas reconcile their logs. An old leader may have an uncommitted tail that never reached the surviving replicas. On returning as a follower, it must align its history with the current leader, which can require truncating a divergent tail before fetching more records.

This is why having the largest local log does not automatically make a replica the correct leader. A replica needs to follow the partition history that the current leader maintains; Kafka does not merge conflicting tails from several former leaders.

The committed batch in our successful takeover example is already present on Broker 2. An earlier acks=1 write that existed only on the failed leader would have a different risk: the new leader might never have received it.

Uncertain Producer Outcomes

A leader can store and replicate a batch, then fail before its successful response reaches the producer. The producer sees a timeout even though the batch exists on the new leader.

Producer idempotence helps prevent duplicate log entries from the producer’s own retries. An application that submits a fresh event after an uncertain result still needs to decide whether that event represents repeated business work. Leadership recovery cannot resolve that application decision.

Kafka’s topic description exposes the current leader and replica membership for each partition. With the Kafka 4.3 command-line tools and an existing topic on a local learning cluster reachable at localhost:9092, you can inspect it with this read-only command:

For a secured cluster, use a reachable bootstrap address and the appropriate client settings through --command-config. The command reports the actual cluster state; it does not create this lesson’s three-broker example.

After Broker 1 has returned and caught up, the relevant fields for our partition could have these values. This table interprets an illustrative result rather than reproducing the full command output.

The leader field is decisive for current leadership. Broker 1 can remain first in the ordered replica list, making it the preferred leader, while Broker 2 is the actual leader.

The ISR field also has a separate purpose from the replica list. An assigned follower can temporarily fall out of sync while remaining assigned to the partition. A topic description is a useful state snapshot, but understanding why a follower is behind requires replication and broker metrics over time.

Each available partition has one current leader that accepts producer writes and establishes log order. Followers actively fetch and copy that history, retaining the same offsets.

A local append, follower progress, the replication high watermark, and consumer progress are distinct states. Their separation explains why a record can exist on a broker before it becomes readable or before an application processes it.

Leadership can move to another replica. Metadata and leader epochs help participants follow the current authority, while log reconciliation brings returning replicas into agreement with the new leader.