The New Consumer Rebalance Protocol

In a classic consumer group, a membership change brings consumers together for another join and synchronization round. Cooperative assignment can reduce unnecessary partition revocations, but the group still depends on that shared coordination process.

The newer consumer rebalance protocol changes the process itself. The broker coordinator computes assignments, and consumers move toward them through individual heartbeat exchanges. A partition transfer can proceed without making every member complete a new group-wide join round.

In this chapter, we’ll follow that process from joining to recovery, then examine the configuration and migration changes it introduces. We’ll use a KRaft-based Kafka 4.3 cluster and the Apache Kafka Java consumer with automatic assignment through subscribe(). The protocol, introduced by KIP-848, has been generally available since Kafka 4.0. In the Kafka 4.3 Java client, using it still requires setting group.protocol=consumer; the default remains classic.