For many years, running Kafka also meant running ZooKeeper. Brokers stored application records, while ZooKeeper held cluster metadata and helped Kafka coordinate which broker acted as controller.

That arrangement supported Kafka through years of growth. As clusters accumulated more partitions and operational responsibilities, maintaining metadata across ZooKeeper, the Kafka controller, and the brokers became harder to manage.

Kafka replaced this architecture with KRaft, its Raft-based metadata system. To understand why, we’ll look at the work ZooKeeper performed, the limitations of Kafka’s integration with it, and what changed when Kafka took responsibility for its own metadata. ZooKeeper examples here describe historical deployments; current Kafka 4.x deployments use KRaft.

Apache ZooKeeper is a distributed coordination service. It stores small pieces of shared state and provides mechanisms that applications can use for membership tracking, configuration, and leader election. An ensemble is a group of ZooKeeper servers working together.

ZooKeeper organizes its data into a tree of znodes, entries with paths. Some entries persist until a client deletes them. Others are ephemeral, meaning ZooKeeper removes them when the session that created them expires.

In ZooKeeper-based Kafka, brokers used ephemeral registrations to indicate their presence. A broker maintained a session with ZooKeeper, and losing that session eventually removed its registration. Kafka could then react to the membership change. A brief network interruption did not necessarily expire a session immediately; failure detection depended on session timing.

Kafka also used ZooKeeper to store information such as topic definitions, replica assignments, partition state, and configuration. One Kafka broker took on the additional role of active controller, using ZooKeeper to coordinate that election.

The Kafka controller made decisions such as selecting replacement partition leaders. ZooKeeper provided shared state and basic coordination operations; it did not apply Kafka’s rules for choosing which partition replica should lead.

The diagram shows the main relationships in a historical ZooKeeper-based cluster. Controller work ran inside one of the broker processes.

Application records traveled between clients and brokers. ZooKeeper did not receive every order event or approve each produce request. Clients could also contact Broker 1 for the partitions it served; the diagram separates one client path to keep the coordination relationships readable.

For the modern Java consumer used in the later ZooKeeper era, group coordination and committed offsets already lived in Kafka. Much older consumer implementations used ZooKeeper for those responsibilities. Removing ZooKeeper was therefore not the event that first moved modern consumer offset storage into Kafka.

Persisting metadata is only part of the problem. Kafka also needs the controller and brokers to act on the appropriate state.

In the ZooKeeper architecture, persistent metadata lived in ZooKeeper, the active controller maintained an in-memory view, and brokers maintained their own local views. Kafka had to keep those views synchronized through ZooKeeper notifications, reads, and controller-to-broker requests.

A ZooKeeper watch lets a client receive a notification about a relevant change. The traditional watches Kafka used were one-shot notifications: after a watch fired, the client needed to read the state and arrange further notification. A notification told the client that something changed; it did not supply a replayable stream of every Kafka metadata update.

ZooKeeper itself orders its updates and provides coordination guarantees. The difficulty was that Kafka’s complete metadata-management workflow extended beyond that storage system. Persisting a change, updating the controller’s view, and informing every affected broker were separate pieces of work.

Consider orders.placed, where Broker 1 leads partition 0 and Brokers 2 and 3 hold its follower replicas. Suppose Broker 3 becomes unavailable. Kafka needs its membership view and partition state to reflect that change, and the relevant brokers need to learn the result.

If a notification or controller request arrives late, a component can temporarily retain an older view. Kafka used versions, retries, and reconciliation to handle such situations. Each additional update path made that recovery logic harder to reason about.

The exact paths evolved across Kafka releases. Earlier implementations allowed more direct ZooKeeper access from brokers and administrative tools; later work moved operations behind Kafka APIs. The broader architectural problem remained: Kafka needed a dependable way for each participant to identify and apply the metadata changes it had missed.

An ordered metadata log makes missed changes easier to track. A participant can track its position in the log and continue from there. This was one of the central motivations for redesigning metadata management.

The old controller role also affected recovery. When the active controller failed, another broker had to win the controller election and initialize its controller state from ZooKeeper. The new controller had to load cluster metadata before it could manage the cluster.

Suppose Broker 1 both leads orders.placed partition 0 and acts as the cluster controller. If its process crashes, Kafka loses both its partition leader and its active controller.

Assume ZooKeeper still has a working majority and Broker 2 has a replica eligible to lead partition 0. A replacement controller must become ready before it can finish coordinating the partition’s recovery. Broker 2’s copy of the records is necessary, but metadata management must also recover.

The diagram shows the dependency between those steps. It does not assign a fixed duration to any of them.

Partitions whose leaders remained available could continue serving some traffic during controller recovery. Losing the controller did not instantly erase partition data or require every healthy partition to change leaders.

The amount of metadata mattered, though. Imagine a platform with 2,000 topics, each containing 50 partitions. That is 100,000 partitions whose assignments and leadership Kafka must track, before considering configuration and ongoing changes. This is an illustrative workload, not a stated Kafka capacity limit.

Even if those topics receive little traffic, the cluster still has substantial metadata to load, distribute, and maintain. Topic creation, replica reassignment, and broker failures add further control work. More partitions can therefore increase recovery and administration costs without increasing record throughput.

Adding brokers gives the cluster more resources for storing and serving records. It does not automatically remove the work of rebuilding controller state. Kafka needed metadata recovery that depended less on loading the full cluster state after each controller election.

ZooKeeper introduced a separate operational responsibility alongside Kafka. Teams had to deploy its ensemble, maintain its storage, monitor its health, secure its connections, and account for its behavior during upgrades and failures.

Those responsibilities were related to Kafka operations but required different configuration and tools. Securing producer-to-broker connections, for example, did not automatically secure broker-to-ZooKeeper connections.

Consider a maintenance window in which Kafka brokers appear healthy but ZooKeeper cannot form a majority. Existing partition leaders may continue handling some requests for a time. Operations that depend on coordination, including controller election, cannot proceed normally. The operator has to investigate both systems to understand the cluster’s condition.

ZooKeeper was a replicated service, so describing it as one unprotected server would be inaccurate. The concern was having to operate another distributed system and handle failures in its communication with Kafka.

Replacing ZooKeeper also let the Kafka project develop and test one metadata architecture. Maintaining several alternative coordination backends would retain much of the deployment burden and expand the set of combinations that needed testing. A Kafka-managed metadata system gave the project control over both the metadata representation and its integration with brokers.

KRaft brings metadata storage and controller coordination into Kafka. Controllers form a metadata quorum, a group that uses Raft consensus to agree on an ordered log of metadata changes. One controller is active; the others replicate the log and maintain state so they can take over.

This changes how the cluster records updates, distributes them, and prepares for controller failure. The diagram separates controller replication from the metadata that brokers fetch.

The arrows represent information flow. Standby controllers and brokers request updates; the active controller does not push application records through the quorum.

An Ordered History of Changes

Metadata changes enter a shared log. Brokers apply committed updates in log order and track their progress. If a broker falls behind, it can fetch later changes; snapshots provide a starting state when replaying the retained log alone is insufficient.

Suppose a broker has applied metadata through position 800 while the committed log has advanced to 805. Those positions describe the gap it needs to close. They are metadata-log positions, unrelated to the offsets of order events in orders.placed.

This does not make every broker’s view change simultaneously. It gives brokers a defined history to follow and a way to catch up after delays.

Prepared Replacement Controllers

Standby controllers maintain metadata state while the active controller is running. A replacement therefore does not start with the same full reload from an external ZooKeeper store that the old architecture required.

Election, catch-up, and activation still take time. A controller recovering from disk also needs to load its state. The improvement is that a running standby already participates in metadata replication, reducing the work needed when leadership changes.

Independent Controller Deployment

KRaft allows dedicated controller processes, so operators can separate metadata management from broker workloads. A broker restart then need not also remove a controller process, and controllers can have their own resource allocation and maintenance schedule.

Kafka also supports combined broker and controller processes, which are convenient for local learning. In that arrangement, the two roles still share a process and its failure risks. Removing ZooKeeper does not necessarily reduce a production deployment to fewer machines; dedicated Kafka controllers take over the coordination responsibility.

The table summarizes the main architectural differences.

These changes address metadata management and recovery. They do not imply a universal multiplier for producer throughput. Record performance still depends on broker storage, networking, batching, replication, and the workload.

Kafka introduced KRaft gradually. Kafka 3.3 marked it production-ready for new clusters. Migrating existing ZooKeeper-based clusters was a separate capability that matured afterward.

Kafka 3.9 was the final release line supporting ZooKeeper mode. Kafka 4.0 removed both ZooKeeper mode and the ZooKeeper-to-KRaft migration functionality. An existing ZooKeeper cluster must therefore complete its migration on a supported earlier release before upgrading to Kafka 4.x.

For example, Kafka 3.9 can serve as a bridge: upgrade to the appropriate 3.9 release, complete the supported metadata migration, then proceed to 4.x. The actual path depends on the starting Kafka and ZooKeeper versions. This sequence explains the compatibility boundary; it is not a complete migration procedure.

Migration involves transferring metadata and changing which system manages it. Simply deleting ZooKeeper configuration does not convert an existing cluster. Migration also has a finalization boundary after which Kafka no longer supports reverting to ZooKeeper, so operators need to follow the procedure for their exact release.

For applications using supported Kafka client protocols, the basic interface remains familiar: producers write records to brokers, and consumers fetch them. Tools that directly accessed ZooKeeper require replacement with Kafka APIs or supported administrative commands. Client-version compatibility and other changes in a major Kafka upgrade still need separate attention.

KRaft retains the need for a healthy coordination majority. A quorum with three voting controllers needs two to make progress. If it loses that majority, metadata changes cannot commit, even if several brokers and their disks remain healthy.

Operators must still protect controller storage, provide reliable connectivity, monitor metadata progress, and plan recovery. Applications must still handle retries and repeated processing. The metadata redesign does not make a Kafka write and an external database update atomic, or guarantee that every partition always has an eligible replacement leader.

Kafka moved away from ZooKeeper to simplify operations and redesign how Kafka maintains cluster metadata. The earlier architecture spread that work across an external store, an elected broker controller, and broker-local views. Large metadata sets and controller recovery made the costs of that arrangement more apparent.

KRaft uses an ordered metadata log, continuously maintained standby controllers, and Kafka-managed coordination. These changes simplify metadata synchronization and recovery. Operators still need to keep quorums healthy and partition replicas available.