Starting another instance of a Kafka consumer can spread the work across more machines. But the consumers still need to agree on which partitions each instance should read. If one instance disappears, the others need to take over without relying on the failed instance’s memory.

Consumer groups solve this through membership, partition assignments, and saved progress. In this chapter, we’ll follow a reporting application from joining a group to recovering after a failure, and see how the client and brokers divide those responsibilities.

We’ll assume a KRaft-based Kafka 4.3 cluster and the Apache Kafka Java consumer using subscribe() for automatic partition assignment. We’ll distinguish the classic and consumer group protocols where their internals differ. This discussion concerns regular consumer groups, not share groups.

Suppose an order-reporting application reads an orders.placed topic with four partitions. Two consumer instances, A and B, both subscribe with group.id=order-reporting.

The shared group ID identifies their common pool of work. Each consumer also has a member ID, which identifies that participant in the group protocol. A member is a consumer instance, not necessarily an entire application process: one process can host multiple consumers.

Internally, the group needs to keep track of several related pieces of state:

These pieces answer different questions. An assignment tells A that it may read partition 0. A committed offset tells a reader where the group last saved progress in that partition. Neither tells Kafka whether a reporting database update succeeded.

Once the assignment settles, each partition has at most one assigned member within this group. A consumer may own several partitions. For example, A might own partitions 0 and 1, while B owns 2 and 3. The exact result depends on the assignment strategy.

With four partitions, at most four members can hold partition assignments for this topic at once. A fifth member would have no partition to read if this is the group’s only topic. Kafka divides this work by partition; it does not give different records from one partition to different members of a regular group.

A separate group, such as order-notifications, has its own assignments and committed offsets. Its consumers can read the same stored order events independently. Kafka does not make another copy of the topic for each group.

The consumers need a common authority for group membership. Kafka assigns that responsibility to a broker acting as the group coordinator. All members of order-reporting communicate with that coordinator for group management and Kafka-managed offset operations.

The coordinator is a broker role, separate from the KRaft controller’s cluster metadata work. It may also lead ordinary topic partitions, but coordinating a group does not make it the leader of every partition that group reads.

This gives each consumer two kinds of communication: group requests to its coordinator, and fetch requests to the brokers serving its assigned partitions. The diagram shows A’s connections, assuming the usual configuration in which consumers fetch from partition leaders.

The coordinator does not forward order records to A. A requests them directly, including the partition and offset it wants to read. The group also does not need a coordinator decision for every record; its assignment remains useful across many fetch requests.

Kafka stores durable group metadata and committed offsets in the internal __consumer_offsets topic. A group ID maps to one partition of that topic, and the broker leading that internal partition acts as the group’s coordinator. Many groups can share the same internal partition and coordinator.

This placement gives group state a recovery path through Kafka’s replicated storage. It also explains why the coordinator can change even when the application’s group ID stays the same.

When A starts, it uses bootstrap.servers to discover the cluster. It can send a FindCoordinator request to a broker to locate the coordinator for order-reporting. The bootstrap broker and the coordinator need not be the same broker.

Calling subscribe() declares the topics A wants. It does not mean A already owns their partitions. As the consumer runs its poll loop, the client completes group coordination, receives an assignment, establishes starting positions, and begins returning records.

The way members agree on an assignment depends on the group protocol. An assignor is the component that calculates how to divide eligible partitions among members.

The Classic Protocol

With group.protocol=classic, members send JoinGroup requests containing their subscription and supported assignment information. The coordinator chooses a compatible assignment protocol and selects one member as the group leader.

This leader is a consumer, not a broker or the KRaft controller. It receives the group’s membership and subscription information and runs the selected assignor. It then submits the resulting assignments through SyncGroup. Other members also send SyncGroup requests, and the coordinator returns each member’s assignment.

The diagram shows this exchange for A and B, with the coordinator choosing A as group leader.

The coordinator manages the membership round and distributes the result; the selected consumer computes the assignment. Afterward, the group leader reads its own partitions just like any other member. Other consumers do not fetch records through it.

The Consumer Protocol

With group.protocol=consumer, a server-side assignor calculates the target assignment at the coordinator. Members use ConsumerGroupHeartbeat requests to join, report their state, and receive assignment changes. There is no consumer group leader computing assignments.

Members move toward the target assignment incrementally. Before another member can take over a partition, its current owner must release it or the coordinator must remove that owner. An intended assignment and a member’s current assignment can therefore differ while the transfer is in progress.

The consumer protocol has been generally available since Kafka 4.0. In the Kafka 4.3 Java client, classic remains the default; using the newer protocol requires setting group.protocol=consumer.

Both protocols establish partition ownership, but they reach that agreement differently. The important result for A is the same: it learns which partitions it should read before normal consumption proceeds.

Suppose the assignment gives A partitions 0 and 1, and B partitions 2 and 3. Each consumer maintains a local position for each assigned partition. In the Java consumer, this position advances as poll() returns records to the application.

That position can run ahead of completed business work. If a poll returns ten records, the application may still be processing the first one even though the consumer’s position has advanced past the batch.

A committed offset is different: it records where the group should resume. Conceptually, Kafka identifies a commit by the group ID, topic, and partition. It is not tied to whichever member happened to make the commit.

Consider partition 0. Assume the application has processed every earlier record, and the group has successfully committed 120. A now receives records at offsets 120 through 129. After the reporting database successfully handles all ten records, A can commit 130 as the next position from which to resume.

The diagram shows the interval after processing succeeds but before the new commit succeeds. Each other partition has its own progress, which this diagram leaves out.

The database is ahead of Kafka’s saved checkpoint. If A disappears now, its replacement will normally resume at 120, assuming that offset and the records are still available. The replacement does not inherit A’s in-memory position of 130.

This example assumes explicit commits after successful processing. Enabling automatic commits does not teach the client when an external database update has finished. The application must choose a commit strategy that matches how it processes records.

When there is no valid committed offset, the consumer uses its configured auto.offset.reset behavior, unless the application explicitly chooses a position. That may mean starting at the earliest available offset, starting at the end, or failing. A rebalance alone does not reset the group’s progress.

Assignments cannot remain fixed forever. A third consumer may join, B may shut down, or A may stop communicating. Kafka needs to update responsibility for the affected partitions. This reassignment process is a rebalance.

A heartbeat lets the coordinator know that a member is still participating. If the coordinator does not receive the required communication before the session expires, it can remove the member. This is a timeout-based decision: the coordinator cannot directly distinguish a crashed process from one that a network failure has isolated.

The Java client also limits how long the application can go between poll calls through max.poll.interval.ms. This helps detect a consumer that remains connected but is not returning to its poll loop. A live network connection does not prove that application processing is making useful progress.

For an orderly partition transfer, the current owner stops taking on work for the partitions it must release, handles outstanding work according to the application’s policy, and relinquishes them. The replacement establishes its own starting positions and begins reading. The protocol and assignment strategy determine how much other work must pause during this change.

Membership requests carry version information so an old participant cannot keep acting as though nothing changed. The classic protocol uses a group generation, a number identifying a membership round. The consumer protocol uses epochs, version numbers for group, assignment, and member state. These let the coordinator detect stale membership information and reject invalid requests, including offset commits that fail membership validation.

This protection has a boundary. The coordinator cannot cancel a SQL statement already running in A’s application or stop a detached worker from writing to a database. Applications that hand records to worker threads must also manage that work when partition ownership ends.

For example, if A loses partition 0 while a worker is still updating an order, the new owner may begin handling the same event. Group coordination alone cannot make those external operations safe.

Return to the reporting group with A owning partitions 0 and 1, and B owning 2 and 3. Suppose A crashes at the point shown earlier: it has processed partition 0 through offset 129, but the last successful commit is still 120.

After the coordinator detects A’s failure, the group can assign A’s partitions to B. This is one possible recovery outcome with B as the only remaining member. The diagram focuses on partition 0; B also needs to establish a position for partition 1 from that partition’s own saved progress.

B does not request A’s buffered records or local memory. It reads the partition log from Kafka using the group’s saved offset. That is why recovery works even when A’s machine is gone, provided the required Kafka state and records remain available.

It is also why the application can process records twice. If offset 124 contains event evt-8124 for order ord-5204, B will encounter that event again. A reporting operation that blindly increments a total could count the order twice. One possible application design stores the event ID and the reporting update in the same database transaction. A uniqueness constraint prevents the application from applying that event again.

Committing earlier would change the failure outcome. If A had committed 130 before completing the database updates, B would resume at 130 and could leave unfinished work behind. Kafka would still retain the earlier records according to topic policy, but normal recovery from the saved checkpoint would skip them.

The same separation matters when the coordinator broker fails. Once another eligible replica leads the relevant __consumer_offsets partition, that broker can load the group’s durable state and become its coordinator. Clients rediscover the coordinator and retry group operations as needed. Coordinator recovery restores coordination and saved checkpoints; it does not recover business updates that the application never completed.

A consumer group combines membership, partition ownership, and saved offsets. A broker coordinator manages the group, while consumers fetch records directly from the brokers serving their assigned partitions. Classic groups compute assignments in a selected consumer; the consumer protocol moves that work to the coordinator.

When ownership changes, a replacement can resume from the group’s committed offset without needing the previous consumer’s memory. That enables recovery, but the checkpoint can lag behind completed processing or get ahead of unfinished work. Correct application behavior depends on keeping commits aligned with processing and handling repeated external effects safely.