A producer is sending order events to a partition on Broker 1. When that broker fails, another replica may take over. The producer needs to discover the new leader, and the surviving brokers need to agree about which replica now has that responsibility.

Kafka uses cluster metadata to describe the cluster and coordinate changes to it. This information connects a topic name in your application to the brokers that store and serve its partitions.

This chapter explains what metadata contains, how Kafka maintains it, and how clients use it during normal operation and recovery. The examples assume a KRaft-based cluster and standard Kafka producers and consumers, with consumers fetching from partition leaders.

Cluster metadata describes Kafka's structure and shared configuration. It answers questions such as which brokers have registered, which topics exist, where their replicas belong, and which broker leads each partition.

Consider a topic called orders.placed with three partitions and a replication factor of three. In a cluster with Brokers 1, 2, and 3, its partition metadata might look like this:

Each partition has one leader and three assigned replicas, including the leader. The in-sync replica set, or ISR, identifies replicas Kafka considers sufficiently caught up with the leader. Here, every assigned replica is in sync.

Assignment and synchronization describe different things. If Broker 3 falls behind on partition 0, it can remain an assigned replica while Kafka removes it from that partition's ISR. The assignment says where a copy belongs; the ISR tracks which copies currently meet Kafka's synchronization requirements.

The cluster also needs information beyond this table:

For example, the topic's retention override belongs to shared configuration. An individual record's timestamp belongs to that record. Both affect how Kafka behaves, but they live in different places.

Kafka does not store every setting in cluster metadata. Some settings come from a server's local configuration, and producer settings such as batch.size belong to the application. Keeping those scopes separate helps when deciding which component needs a configuration change.

Suppose partition 0 contains an order event with key ord-1042 at offset 42. The key, value, headers, and timestamp are part of the record in the partition's data log. Cluster metadata describes which brokers hold that log and which one leads it.

Appending another order usually changes the data log without changing the partition's assignment or leader. Moving leadership from Broker 1 to Broker 2 changes cluster metadata without moving every record through the controller.

The diagram separates these responsibilities. The arrows show which information each component uses, rather than a sequence of requests.

The controller does not handle each order event. Brokers use their metadata view to perform the reads, writes, and replication for the partitions they host.

Consumer and Transaction State

The word “metadata” also appears in discussions of consumer groups and transactions. Distinguish those uses from the cluster metadata that KRaft maintains.

Kafka persists consumer group state and committed offsets in __consumer_offsets. Transaction coordinators persist transaction state in __transaction_state. These are internal topics whose partitions reside on brokers.

Cluster metadata describes the partitions of these internal topics, just as it describes the partitions of orders.placed. The contents of those partitions remain separate. Committing the reporting group's next offset as 43 does not add that offset commit to the KRaft metadata log.

This distinction also separates responsibilities: the controller manages partition ownership, while broker coordinators manage groups and transactions.

In KRaft, controllers maintain a replicated metadata log, an ordered history of changes to cluster metadata. One controller is active at a time. It handles metadata changes, and the controller quorum replicates the log so that committed changes can survive controller failures.

A quorum is the set of voting controllers that participates in this agreement. Committing changes requires a majority. With three voting controllers, two must be available and able to communicate for the quorum to make progress.

The metadata log contains records describing changes such as registering a broker, creating a topic, or updating a partition's leader. Kafka applies these records to build the current metadata state. A metadata snapshot captures that state at a point in the log, allowing recovery without replaying the entire history from the beginning.

Brokers fetch metadata updates and maintain local views. Clients obtain the routing information they need from brokers. These are different views of the same cluster, with different amounts of information.

The diagram shows how a committed change reaches an application. The broker fetches updates, and the client requests metadata; the arrows indicate the direction in which the information travels.

This arrangement lets a broker answer metadata requests using its local state. Producers can send batches without first contacting the active controller.

It also means a committed change does not become visible everywhere at the same instant. A broker may still be applying an update while a client holds an older response. For a short period, two clients can have different views of a partition's leader.

Kafka therefore needs both an authoritative sequence of metadata changes and mechanisms for detecting stale information. Replicating the metadata log establishes the sequence; refreshing client metadata helps applications catch up with it.

A Kafka producer commonly starts with configuration like this:

These are illustrative broker addresses. A real application also needs the security settings required by its cluster.

bootstrap.servers gives the client initial contact points. It does not pin all requests to those brokers or list every broker the client will ever use. Multiple addresses help the client establish its initial connection when one address is unavailable.

Suppose the producer initially contacts Broker 2. It requests metadata for orders.placed and learns which brokers lead the topic's partitions, along with the advertised endpoints needed to reach them.

When the producer selects partition 0 for an order event, it connects to Broker 1, the leader in our example. Broker 2 does not forward the produce request just because it supplied the metadata. The producer can contact Broker 3 for partition 2 even though Broker 3 was absent from the bootstrap list.

A consumer uses partition metadata to locate the brokers serving its assigned partitions. It discovers its group coordinator through a separate coordinator lookup. Partition leadership and group coordination can point to different brokers.

Metadata Responses and Refreshes

A client's Metadata response describes the brokers and partitions it needs to route requests. It does not deliver the full controller metadata state, all topic configuration, or consumer group offsets. Administrative tools use other APIs to inspect those details.

Kafka clients cache metadata to avoid fetching it for every request. In Apache Kafka's Java clients, refreshes can follow events such as discovering an unknown topic or receiving an error indicating that leadership information is stale. Periodic refreshes also help clients discover changes; metadata.max.age.ms controls the maximum age before a forced refresh.

The exact refresh and retry behavior depends on the client and version. The periodic refresh interval is not a guarantee about failover time, because clients can refresh earlier and recovery also depends on the cluster completing the leadership change.

Advertised Broker Addresses

Discovery succeeds only if the returned broker addresses work from the application's network. Kafka uses advertised.listeners to publish endpoints, which may differ from the addresses on which a broker binds its sockets.

For example, a laptop might reach a bootstrap endpoint through a published container port, then receive broker-2:9092 in metadata. If broker-2 resolves only inside the container network, the laptop cannot establish the next connection.

The initial connection can succeed while subsequent produce or fetch requests fail. Inspect the returned endpoints and check DNS, routing, ports, and security from the client's environment. Adding more bootstrap addresses will not fix an unreachable endpoint that the partition leader advertises.

Return to partition 0, with Broker 1 as leader and Brokers 2 and 3 as in-sync followers. Assume Broker 1 fails, the controller quorum remains available, and Broker 2 is eligible to become leader.

The controller detects the broker's unavailability through broker liveness tracking. It chooses a replacement and records the partition state change through the metadata quorum. Brokers then apply the committed change, and clients learn the new leader through updated metadata.

For this example, suppose the partition's leader epoch changes from 7 to 8. A leader epoch is a version number for a partition's leadership. It helps Kafka distinguish requests or information associated with older leadership from the current state. It is separate from a record offset and from the controller quorum's own election epoch.

The sequence below follows one producer through the transition. Its metadata remains stale until it refreshes.

During the transition, the producer may see a connection failure, a timeout, or a leadership-related error. It can even refresh before the contacted broker has applied the new state, requiring another attempt. Recovery takes time: Kafka must detect the failure and commit the metadata change. Brokers then apply it, and clients discover the new leader.

Changing the leader does not necessarily change the replica assignment. The assignment can still include Broker 1 while it is unavailable. If it returns, it must recover and catch up before Kafka treats it as in sync again.

There is also a separate question about the last write. Broker 1 might have stored and replicated an order before failing, while its acknowledgment never reached the producer. Discovering Broker 2 does not tell the application whether that particular write succeeded. Producer idempotence and retry handling address that ambiguity; refreshing metadata alone does not prevent duplicates.

If no suitable replica can lead, refreshing metadata cannot make the partition available. If the controller quorum loses its majority, it cannot commit the leadership change. Existing leaders may temporarily serve requests, but the cluster still needs a controller majority to recover from further failures or change its structure.

You can inspect partition placement without opening controller log files. The following read-only commands assume an Apache Kafka 4.x distribution, a running KRaft cluster, and an existing orders.placed topic. Run them from the distribution directory and replace the broker address with one reachable from your environment.

For a secured cluster, add --command-config client.properties with the required authentication and TLS configuration. Your principal also needs permission to describe the requested resources.

Start with the topic description:

The output reports partition leaders, assigned replicas, and ISR membership. The table below extracts those fields from an illustrative result after Broker 1 has failed and leadership has settled. It is not captured output from a live cluster.

Broker 1 remains in the replica assignments but is absent from the ISR sets. Partition 0 has a replacement leader. This is evidence of the cluster's reported partition state, not a count of retained records or a measurement of consumer lag.

To inspect the metadata quorum itself, use:

Both examples contact Broker 2 so they can run while Broker 1 is unavailable. The quorum command reports information such as the cluster ID, metadata quorum leader, leader epoch, and high watermark. Here, the high watermark concerns committed progress in the metadata log. It does not describe committed order records in partition 0.

The two commands answer different operational questions. A topic description helps explain where application requests should go and which replicas are in sync. A quorum description helps inspect the controllers responsible for metadata progress. Neither command alone proves that an application can resolve every advertised endpoint, authenticate, or complete a write.

Cluster metadata describes broker membership, topic identities, partition ownership, and shared configuration. KRaft maintains the authoritative history of changes, brokers apply those changes locally, and clients cache the routing information they obtain from brokers.

That separation allows normal record traffic to proceed directly between clients and brokers. It also explains why recovery involves both changing cluster state and refreshing stale client information. When diagnosing a problem, distinguish partition availability, metadata quorum progress, and the client's ability to reach the advertised brokers.