In this lab, you run a three-node KRaft cluster and inspect its metadata quorum. You observe leader election, follower catch-up, broker registration and fencing, and the effects of losing a majority. You use only the tools bundled in the apache/kafka:4.3.1 image.

Use this fixed lab setup: the cluster has voters 1, 2, and 3, every node runs both roles, and the only topic you create before the failure phases is orders.placed with three partitions and replication factor 3. That gives one topic record, three partition records, one leader epoch before the failover and a larger one after it, and log end offsets that must agree across the voters once lag drains.

Environment: E2, the three combined broker-and-controller nodes kafka-1, kafka-2, and kafka-3. Docker and Docker Compose run on your host. Every Kafka tool runs inside a container through docker exec with a bootstrap address of kafka-1:9092, kafka-2:9092, or kafka-3:9092; always bootstrap through a node that is running. Keep the Compose file, decoded logs, observations, and report on the host. Run docker compose commands from kafka-labs/cluster/ and everything else from kafka-labs/metadata-quorum/.

Create:

Reuse the cluster compose file from the brokers module lab after completing that lab’s cleanup. Start with fresh lab storage and only the three original services. Before the first startup, add KAFKA_METADATA_LOG_MAX_SNAPSHOT_INTERVAL_MS: 60000 under each service’s environment to shorten the snapshot wait for Phase 6. Create the observations/ and decoded/ directories before saving outputs. From kafka-labs/cluster/, start all three nodes:

Run the topic-list command below against kafka-1:9092. With fresh lab storage, expect an empty result and exit status 0. A successful listing confirms that this endpoint answers requests; inspect the quorum and brokers before the failure phases.

Host clients would use localhost:19092,localhost:29092,localhost:39092; this lab needs none.

Describe the quorum status, then the per-replica replication view, and save both outputs as observations/01-baseline.txt:

Write down, in your own table: the leader id, the leader epoch, the high watermark, and for each of nodes 1, 2, and 3 the LogEndOffset, Lag, and Status columns.

What you should see: CurrentVoters lists exactly 1, 2, and 3. One node has Status Leader and the other two are Follower; that node's id matches LeaderId. Once the followers have caught up, all three LogEndOffset values are equal and every Lag is 0. The high watermark is close to those offsets; if it is smaller, re-run the command and watch it catch up. The log is not empty on a fresh cluster, because the quorum committed broker registrations and feature records during startup. Record whether CurrentObservers lists anything.

Create the topic:

Immediately re-run describe --replication and describe --status, then describe the topic. Save all three outputs as observations/02-after-create.txt:

Now read the metadata log itself. Each node stores the metadata partition under its log directory. Decode the available segments on kafka-1. Find the TOPIC_RECORD for orders.placed, note its topicId, then copy that record and the three PARTITION_RECORD entries with the same ID into decoded/topic-records.txt:

Run the same decode on kafka-2, saving it under a different filename, and compare the matching topic records. Filtering by record type alone would also include other topics. On fresh lab storage these segments should still contain the creation records; if they do not, record that retention removed the history rather than guessing a segment filename.

What you should see: Compare each controller’s log end and the high watermark with Phase 1. Decode the topic-creation records to identify their offsets. Other metadata work can also advance the log, so the total offset change need not equal the number of creation records. decoded/topic-records.txt holds one TOPIC_RECORD naming orders.placed with a topicId, and three PARTITION_RECORD entries with partitionId 0, 1, and 2, each carrying replicas, isr, and leader. The topicId matches the TopicId in the topic description, and each partition's leader and isr match its row there. The decode from kafka-2 contains the same records at the same offsets, because followers copy the leader's log rather than composing their own. Confirm those offsets sit below the new high watermark.

Take the LeaderId from Phase 2 and call that node L. Note the leader epoch. Then stop L and record the time:

Replace L with the actual id. From a surviving node, describe the status until it reports a leader, and note when it does. The examples use kafka-2; substitute if kafka-2 was L:

Publish three keyed events, then read them back:

Describe the topic through kafka-2 as well. Then bring L back and watch it rejoin:

Run describe --replication through kafka-2 several times over the next minute. Save every output from this phase, with the time you ran each command, as observations/03-failover.txt. You are timing the response to a graceful stop, the election, and the old leader’s return. The optional abrupt-stop experiment lets you compare failure detection without a graceful resignation.

What you should see: the status reports a different LeaderId and a LeaderEpoch larger than the one you noted; record the exact value, since it can grow by more than one if an election attempt failed. The producer exits without error and the consumer prints three records with keys ord-1042, ord-1043, and ord-1044. The topic description shows one of the two running nodes leading each partition with an Isr of two ids, because L also carried a broker role in this combined-mode cluster; that ISR change is partition metadata, not quorum leadership. After L starts, its row normally returns with Status Follower, a LogEndOffset that catches up with the leader’s and Lag that reaches 0. You may miss the brief catch-up interval; record what you observe. Restarting L does not force leadership back to it; note any further elections.

Use kafka-3 unless it is the current leader, in which case use kafka-2 and substitute throughout. The current leader's container log is where the controller records its decisions.

Stop kafka-3, wait at least 60 seconds, then describe the topic and the replication view through kafka-1:

Start it again and record the time, then pull the controller's log lines about broker 3, with kafka-L replaced by the current leader's container:

After a minute, re-run describe --replication and the topic description. Save everything as observations/04-fencing.txt.

What you should see: while kafka-3 is stopped, every partition's Isr drops node 3 while Replicas still lists it; assignment and synchronization are separate facts. In the replication view, node 3's LastFetchTimestamp stops advancing and its Lag grows as the remaining quorum commits ISR shrink records without it. After the restart, the controller log shows, in order, a line registering broker 3 again, its fenced state, and a line granting its unfence request because the broker caught up with the offset of its own registration record. Then Isr returns to three ids on every partition. Record both timestamps: unfencing is a broker-level milestone, and rejoining the ISR is a per-partition milestone that can arrive later.

The unfence waited for evidence carried in the broker's own heartbeats: that it had applied the metadata log at least as far as the record that registered it, and that its local startup work was done. A reachable port did not count.

Keep only kafka-1 running; start it first if needed.

Wait 30 seconds, then attempt a metadata change through the survivor. Record the full error text and how long the command took:

Also run describe --status through kafka-1 and record whatever it returns. Then restore the quorum:

First verify that a leader exists and all three voters have caught up. Then list topics again, allowing for broker metadata propagation. If orders.cancelled exists, describe it and verify its partition count and replication factor. If it remains absent, retry creation and verify the result. Save all outputs as observations/05-quorum-loss.txt.

What you should see: with one voter of three, the creation does not succeed. The error names a timeout, because the request waits for a controller that cannot commit. The status command may also time out or show no usable leader; record the exact behavior. After the two nodes return, verify quorum recovery and inspect the topic list. A timeout does not establish whether creation committed. Allow for metadata propagation; if the topic exists, inspect its definition, and if it remains absent, retry creation.

List the metadata directory on kafka-1 and look for files ending in .checkpoint; each name carries the snapshot’s end offset and epoch. Kafka 4.3 defaults to a one-hour snapshot interval when committed metadata lies beyond the latest snapshot; Setup reduces it to one minute. A size threshold can trigger one sooner. If no checkpoint exists, confirm the Setup override and wait after a committed metadata change. Decode an existing checkpoint by replacing SNAPSHOT_FILE.checkpoint with its filename, then read the quorum state file next to it. See the Kafka snapshot configuration for the two triggers.

If a later .checkpoint exists, decode it too and compare record counts. Save the listing and the quorum state as observations/06-snapshot.txt.

What you should see: the directory holds the .log, .index, and .timeindex files of the metadata log plus the .checkpoint files written so far. Its decoded contents are state records such as the topic, partition, and feature-level records, not the full sequence of earlier changes; a snapshot describes the metadata image at its end offset, and the replica next fetches log records starting at that end offset. The quorum state file names the current leader epoch, which equals the latest LeaderEpoch, and the vote this node persisted. That file is why a restarted controller cannot vote twice in one epoch.

From kafka-labs/cluster/, stop and remove the cluster containers and any lab volumes:

Verify that nothing remains:

The listing must contain no containers from this lab’s Compose project. Keep kafka-labs/metadata-quorum/ with your observations and report.

The quorum status and replication views turned the module's vocabulary into numbers you could check: one voter led, the others fetched its log, and the leader advanced the high watermark after learning that replication met the commitment rules. Creating a topic appended a topic record and three partition records that you read back from disk on two nodes at the same offsets, which is what a replicated log means in practice.

After stopping the combined node, you observed a new metadata leader and a higher epoch. You then verified that producers and consumers could work after recovery. You checked the returning node’s role and verified that it caught up. You also traced startup fencing and unfencing, then observed how losing the voter majority prevents new metadata commits. The snapshot and quorum state files are the durable pieces that let those guarantees survive a restart.