When you create a Kafka topic, several servers need to agree that it exists. They need to know its partitions, where the replicas belong, and which brokers should lead them. That information must remain available if the server handling the request fails.

KRaft, short for Kafka Raft, is Kafka’s system for managing cluster metadata through a replicated log. It combines Kafka’s controller logic with a Raft-based consensus protocol, allowing Kafka to coordinate its own cluster state.

We’ll follow a topic-creation request through this system, then examine how brokers use the resulting state and what happens when a controller becomes unavailable. The examples assume a Kafka 4.x cluster with separate broker and controller processes.

KRaft connects two kinds of work. Kafka’s controller logic decides what metadata changes are valid and what those changes should contain. The consensus protocol establishes an agreed history of those changes across controllers.

For example, creating a topic with three replicas requires Kafka-specific decisions about replica placement. Raft does not choose the brokers. It provides the agreement and replication mechanism that protects the resulting metadata.

The controllers form a metadata quorum. One is the active controller, which handles metadata changes. The others follow its log and maintain state so that an eligible controller can take over if needed. Voting controllers participate in elections and in the majority needed to commit changes.

Suppose our cluster has Controllers 101, 102, and 103, plus Brokers 1, 2, and 3. Controller 101 is active. The different ID ranges make the roles easy to distinguish; they are only an example naming choice.

The diagram shows the controller quorum and its relationship with brokers. Arrows between servers indicate information flow; replicas request log updates through fetches.

Brokers obtain metadata from the quorum but do not vote in it when running only the broker role. In replication terminology, they are observers of the metadata log. Adding another broker does not add another controller vote.

A server’s process.roles setting selects the broker role, controller role, or both. Combined processes are useful for local learning, while separate processes allow independent resources and maintenance. The logical responsibilities remain distinct even when they share one process.

The KRaft metadata log records changes to the cluster. It contains information such as broker registrations, topic identities, partition assignments, and configuration updates. Kafka identifies its special metadata partition as __cluster_metadata-0.

Kafka manages this log internally. Applications do not create it with a chosen replication factor or write to it through an ordinary producer. Its voting replicas belong to the controller quorum.

Each metadata record has a position in the log. Applying the records in order produces the current metadata state, a metadata image. The log describes changes over time; the image describes the state after a server applies those changes.

This is the idea behind a replicated state machine: multiple servers apply the same agreed sequence of changes to arrive at the same state. Kafka supplies the rules for interpreting metadata records, and Raft protects their agreed order.

For example, assume orders.placed already exists. Its retention override changes from seven days to one day, and later partition 0 gets a new leader. A server that applies both changes should reach a state containing the new retention override and the new leader.

The diagram illustrates that relationship. The labels summarize changes for teaching purposes; they are not literal on-disk record encodings.

A metadata image does not contain the order events the topic stores. Even changing a retention setting and deleting records according to that setting are separate actions: metadata establishes the policy, and broker log management performs the cleanup.

Kafka also writes metadata snapshots, saved copies of metadata state at specific log positions. A recovering participant can load a suitable snapshot and apply later records instead of replaying every change since the cluster began.

A snapshot therefore preserves the resulting state without needing to retain every intermediate value. It is not a backup of application topic records.

Appending a metadata record on the active controller is not enough to make the change authoritative. That controller could fail before any other controller receives it.

A change becomes committed through the quorum’s replication rules. For a stable quorum of three voting controllers, progress requires a majority of two, including the active controller. The third controller may catch up afterward.

Once the quorum commits a change, a later leader must preserve it, provided failures stay within Raft’s assumptions and the required durable replica state survives. Kafka may discard uncommitted records during recovery.

There are three useful stages to distinguish:

The distinction matters because commitment and visibility are not simultaneous across the cluster. A controller can commit a change while a broker is still fetching or applying it. A client may continue using an older metadata response until it refreshes.

The metadata log’s high watermark marks the boundary of committed progress. It refers to positions in the metadata log. An order record at offset 42 in orders.placed partition 0 has no numerical relationship to metadata-log offset 42.

You also should not equate commitment with protection against every possible loss. Replication tolerates the failures the quorum model covers; destroying the durable metadata copies is a different failure from losing one controller process.

Now consider an administrator creating orders.placed with three partitions and a replication factor of three. Assume the topic does not yet exist, all three brokers have registered and are available, and the administrator has the required permissions.

An ordinary Kafka Admin client connects through broker endpoints. Suppose Broker 2 receives the CreateTopics request and forwards it to active Controller 101. This forwarding applies to the administrative operation; normal produce requests still go directly to partition leaders.

Controller 101 validates the request and determines the topic identity and partition assignments. One possible assignment is:

The table shows a possible result, not a placement guarantee. The controller’s choices depend on the available brokers and placement rules.

The controller generates metadata records describing the topic and its partitions, appends them, and waits for the quorum to commit them before reporting success. Creating a topic can involve several metadata records; it is not necessarily one record per administrative request.

After commitment, two things can proceed: the success response returns to the administrator, and brokers fetch and apply the new metadata. The sequence below shows both in parallel after that point because a successful response does not mean every broker has finished applying the change.

Brokers use the assignments to prepare their local partition logs and take on leader or follower responsibilities. Clients discover the partitions through metadata requests. The Java Admin API explicitly allows a delay between successful topic creation and every broker becoming aware of the topic.

Suppose the administrator receives success and immediately asks another broker to describe orders.placed. That broker may briefly report that it does not know the topic. This can be a propagation delay, so automation should allow bounded retries rather than treating the first response as proof that creation failed.

There is a different ambiguity if the create request times out. The controller might have committed the change while the response never arrived or arrived late. Before issuing another create request, inspect whether the topic exists and whether its configuration matches the intended result. A timeout does not prove that no metadata change occurred.

Once partition 0 is ready, a producer can send an order with key ord-1042 to Broker 1. Storing that order updates the partition’s data log. It does not require a new KRaft metadata record for that individual event.

Maintaining an agreed metadata history is useful only if brokers act on it. A broker registers with the controller and fetches metadata to learn the cluster state. It also sends heartbeats so the controller can track its liveness and readiness.

Metadata fetches and broker heartbeats are separate operations. Fetching transfers the metadata history; heartbeats report information the controller uses to manage broker participation, including metadata progress.

The controller can register a broker but keep it fenced, meaning the controller excludes it from active participation. This is useful during startup: knowing a broker’s ID and address does not mean it has recovered its logs or caught up with metadata enough to serve requests safely.

For example, Broker 3 might restart with a metadata view from before the administrator created orders.placed. It must catch up and complete the required recovery work before becoming available. Otherwise, clients could receive its address while it still lacks the state needed to handle their requests.

Fencing can also follow a loss of broker liveness. It does not delete the broker’s stored records. It changes how the cluster treats that broker until it can participate again.

Distinguish three checks: the process is running, the controller knows about it, and the broker is ready to serve its assigned workload. A successful process-health check alone establishes only the first.

KRaft protects cluster metadata, while topic replication protects application records. A useful cluster needs both, but they have different participants and commitment rules.

In our example, Controllers 101, 102, and 103 replicate metadata. Brokers 1, 2, and 3 replicate the partitions of orders.placed. The fact that both groups contain three servers is incidental.

The controller majority does not determine how a producer’s acks=all request completes. That request follows the partition’s in-sync replica rules and min.insync.replicas requirement. Adding controllers does not increase the replication factor of orders.placed or create additional copies of its records.

A Controller Failure

If Controller 101 fails, Controllers 102 and 103 can elect a replacement while they can communicate and satisfy the election requirements. The replacement continues from the committed metadata history. Clients may need to retry uncommitted work.

This election does not make the new controller the leader of partition 0. Broker 1 can remain that partition’s leader throughout. Administrative requests may pause during controller recovery even while existing partition leaders continue handling record traffic.

Loss of the Controller Majority

If only one voting controller remains reachable, it cannot commit new metadata changes by itself. Topic creation and required partition-leadership changes cannot proceed normally.

Some traffic to existing partition leaders may continue temporarily. That is not a guarantee of normal cluster operation: broker liveness, metadata changes, and further failures can affect availability. Recovery requires restoring a functioning quorum with the appropriate metadata history.

A Partition Failure

A healthy controller quorum cannot replace missing application data. If all replicas of partition 0 are unavailable, the controller may know exactly where they belong while being unable to make that partition usable.

Similarly, successful metadata recovery does not resolve every uncertain producer acknowledgment or repeated consumer action. Clients still need suitable retries, and applications remain responsible for handling repeated processing and external side effects.

These distinctions help explain why “the controllers are healthy” and “the application can write orders” are separate claims. One describes metadata coordination; the other also depends on partition replicas, network access, authorization, and producer settings.

KRaft combines Kafka’s metadata decisions with a Raft-based replicated log. Controllers establish committed history, brokers apply it to manage their replicas, and clients obtain the routing information they need from brokers.

Appending, committing, and applying a change are distinct stages. Understanding those stages explains delayed visibility, uncertain administrative timeouts, and controller recovery. Metadata replication coordinates the cluster, while topic replication and application behavior determine what happens to the records flowing through it.