In this lab you bring up a three-node KRaft cluster, create a replicated topic, and stop brokers while a producer is writing to it. You watch partition leadership move, the in-sync replica set shrink and grow, and the acks=all write policy hold or reject writes depending on how many replicas are in sync.

The baseline setup is orders.placed with three partitions, replication factor 3, and min.insync.replicas=2. Phase 4 temporarily changes the minimum ISR. Every produce run sends a known number of keyed records, so a count you take afterward either matches or it does not. Record partition state with kafka-topics.sh --describe to check leader, replica, and ISR values. Later steps use separate commands to inspect controller state and configuration.

Environment E2, the three-node cluster kafka-lab defined below. Docker Compose runs on the host; every Kafka tool runs inside a container through docker exec. The shell loops that generate records also run on the host and pipe into the console producer.

This lab uses the console producer for the produce-under-failure phases rather than the Java producer from the producers module. The failover and retry behavior lives in the Java client library that the console producer wraps, so the evidence is the same, and you do not need to compile anything. Pass producer settings such as acks and retries with --command-property.

Run every command in this lab from kafka-labs/survive-a-broker-failure/. The docker compose commands point at the cluster file with -f ../cluster/docker-compose.yml.

Create kafka-labs/cluster/docker-compose.yml. Each service is a combined broker and controller, so the three nodes form both the broker set and the three-voter metadata quorum.

Host clients would use localhost:19092,localhost:29092,localhost:39092. The tools in this lab run inside the containers and use kafka-1:9092,kafka-2:9092,kafka-3:9092. Listing all three bootstrap addresses matters here, because a tool you start while one node is not running must still find a live broker.

Start the cluster and confirm all three containers are running:

The diagram maps the phases to the partition states you will record. Follow the ISR value from box to box; the broker numbers are illustrative, and your cluster may place the leader of partition 0 on a different node.

Create the topic with these settings at creation time:

Describe it and save the snapshot:

Each partition line has the shape below. The first broker in Replicas is the preferred leader. On a fresh cluster it should also be the current Leader, and Isr should list all three brokers.

Now identify the active controller:

Record the LeaderId and the current voters. Then set two shell variables in every terminal you use for the rest of the lab. LEADER is the container that leads partition 0 in your snapshot, and TOOLS is any other node; substitute your own values.

What you should see: three partition lines, each with three distinct broker IDs in Replicas, Isr equal to Replicas, and Leader equal to the first replica. The header line shows min.insync.replicas=2 under Configs. The quorum leader is one of nodes 1, 2, or 3, and it may or may not be the same node as LEADER. Write both numbers down: the controller manages leadership metadata, and the partition leader serves writes, and the lab needs you to keep those two roles apart.

The loop below sends 900 keyed records at roughly ten per second, which gives you about ninety seconds to stop a broker while writes are in flight. The producer uses acks=all and raises retries well above the console tool's default of three, so the two-minute delivery timeout, not the retry count, bounds how long a batch may wait for a new leader. It deliberately does not use --sync, so retries happen in the background while new records keep arriving.

About twenty seconds in, stop the leader of partition 0 from a second terminal:

Immediately describe the topic from the second terminal and save the output as observations/phase2-describe.txt using the Phase 1 describe command with kafka-1 replaced by "$TOOLS". Wait for the producer loop to finish, then count what the topic holds:

What you should see: the describe snapshot shows partition 0 with a new Leader and an Isr that has lost the stopped broker. Every partition that had a replica on that node shows a two-member ISR. docker compose stop sends a termination signal, so the broker performs a controlled shutdown and leadership moves before the process exits; the producer log may contain WARN lines naming NotLeaderOrFollowerException while the client refreshes metadata, but it should contain no ERROR line from ErrorLoggingCallback. For a successful run, the count is 900. Also extract and sort the event IDs from observations/phase2-records.txt: verify that every ID from evt-7001 through evt-7900 appears exactly once. A total alone cannot distinguish a missing event from a duplicate. If delivery fails, preserve the errors and record counts and diagnose the failure before repeating on a fresh topic. The consumer prints a timeout error when it runs out of records; that line is expected and is the reason --timeout-ms is there.

Start the stopped node and describe the topic six times, five seconds apart:

The returning broker keeps its container, so it restarts with intact storage. It loads its local logs, registers with the controller, and learns from metadata that it is now a follower of partition 0. It reconciles its log with the current leader using the leader epoch, then fetches from its own log end offset. Those catch-up fetches copy the records that arrived while it was stopped, in this case a slice of the 900 from Phase 2. The leader proposes the ISR expansion only after the follower has reached the high watermark and the start of the current leader epoch.

What you should see: You may capture a two-member ISR before it returns to three. Catch-up can finish before the first sample, and automatic preferred-leader balancing may later restore the original leader. Record what you observe. Restarting restores the replica; a separate election can restore its leadership. Phase 5 examines that distinction. Note the first snapshot showing a three-member ISR.

The goal is an ISR smaller than min.insync.replicas while the cluster can still record the change. Stopping two of three combined nodes would shrink the ISR to one replica, but it would also leave one controller voter out of three, and a lone voter cannot commit the ISR change. The leader would then keep waiting for followers that never fetch, and the producer would see request timeouts instead of the rejection this phase is about. So the lab raises the minimum instead and stops one node.

Raise the topic minimum to three:

Stop LEADER again, then describe the topic and confirm every Isr has two members before producing. Now send three records with acks=all. Set retries to 0 and disable idempotence so the producer reports the first rejection to the callback instead of retrying until the delivery timeout:

Repeat the same command with acks=1 in place of acks=all and append its output to the same log with tee -a. Then run the Phase 2 count command with the output file changed to observations/phase4-records.txt. Separately run grep -c evt-800 observations/phase4-records.txt on that saved record file; do not pipe the first numerical count into another grep.

What you should see: the acks=all run logs three ERROR lines from ErrorLoggingCallback, each naming NotEnoughReplicasException and stating that fewer in-sync replicas are available than required. The acks=1 run logs nothing, because the leader acknowledged its own append without consulting the ISR. The leader has those three records, and the surviving follower may copy them, yet with the ISR below the minimum the high watermark does not advance under the strict minimum-ISR rule, so the count of evt-800 records is 0 at this moment. That is the trade: acks=1 reports success for data that is not yet replication-committed, and a leader failure before the ISR recovers could discard it.

Restore the contract in this order. Start LEADER, wait until describe shows three-member ISRs, then set min.insync.replicas=2 with the same kafka-configs.sh command. Count again and record that the three acks=1 records are now readable.

Describe the topic first. If Leader for partition 0 already equals the first entry in Replicas, Kafka's automatic preferred-leader balancing, which auto.leader.rebalance.enable turns on by default, ran on its schedule before you got here. Record that outcome; the command still applies.

Describe the topic again and append the output to the same file.

What you should see: after the election, Leader for partition 0 equals the first broker in Replicas, and Replicas and Isr are unchanged. If the preferred replica was already leader, the tool reports that the partition did not need an election. Leader placement matters because leaders do the producer-facing work: after two failovers, one node can lead all three partitions while the other two only follow, and a preferred election spreads that work back out without moving a single byte of stored data. The tool also has an option to run the election for every partition at once; check --help before using it on a larger topic.

Show every effective configuration value for the topic, including inherited defaults:

Find the unclean.leader.election.enable line and record its value and its source.

Describe the following hypothetical scenario without performing it on this combined-role cluster. Assume a separate controller quorum remains available. Replica 3 leaves the ISR, then replicas 1 and 2 commit three more records with min.insync.replicas=2. Both safe replicas then become unavailable, leaving only stale replica 3. Assume replica 3 is neither an ELR candidate nor an eligible last known leader. With unclean election disabled, the partition waits for a safe replica. With unclean election enabled, Kafka can promote replica 3 and lose the three committed records it never received. This differs from Phase 4: the rejected acks=all writes there do not establish committed records missing from the stopped replica.

What you should see: the value is false, and its synonyms show that it comes from the default rather than a topic override. Explain the hypothetical scenario in report.md, including why a healthy controller quorum and the absence of safe candidates matter.

Stop and remove the cluster:

Verify that no lab container remains:

The output should list no containers. Keep the observations/ folder and report.md; they are the deliverables. The compose file stays in kafka-labs/cluster/ for later labs.

In a successful run, the consumed records and producer log should account for all 900 submitted events. Check event identities as well as the total before concluding that no acknowledged event was lost. Leadership moved because the controller committed a new leader from the ISR, the client refreshed its metadata and retried, and acks=all meant every acknowledged record already existed on the surviving in-sync replicas. Redundancy came back only after the returning broker had fetched what it missed and passed the ISR admission checks, which explains why a running container may still need time to rejoin the ISR.

The write contract is a relationship between the ISR and min.insync.replicas, not a property of the replication factor. With three assigned replicas and a minimum of two, one failure left writes flowing; raising the minimum to three turned the same failure into NotEnoughReplicasException, and acks=1 sidestepped the check by giving up replication-committed durability. The preferred election restored partition 0 to its preferred leader, unless automatic balancing had already done so, and the unclean election setting stayed false so a stale replica could not rewrite the history the counts had verified.