In this lab, you test a three-node cluster under controlled load. You compare producer tuning settings, inspect consumer lag, observe a broker failure, and move replicas onto a new broker with a throttle. You finish with a rolling restart of every node while a producer runs.

Use these benchmark conditions: the topic is orders.placed with six partitions, replication factor 3, and min.insync.replicas=2. In Phases 1 and 2, each benchmark sends 100,000 records of 1,000 bytes with acks=all. Define matched pairs that differ in one setting. Later failure experiments use different counts and rates. The numbers themselves depend on your machine, which is why you record them instead of predicting them.

Environment: E2. Reuse the cluster compose file from the brokers module lab; Phase 5 adds a fourth service to it. Docker runs on your host. Every Kafka tool, including kafka-producer-perf-test.sh, kafka-consumer-perf-test.sh, and kafka-consumer-groups.sh, runs inside a container through docker exec. Keep the Compose file, reassignment JSON, observations, results, and report on the host. Budget several gigabytes of Docker disk: each Phase 1 or 2 run supplies 100 MB before compression and overhead; replication stores three copies. Open four terminals: Terminal A for producer runs, Terminals B and C for consumer instances, and Terminal D for everything else. Run docker compose commands from kafka-labs/cluster/ and everything else from kafka-labs/tune-and-rebalance/.

Create:

results.md holds the tables for Phases 1 to 3. Paste raw command output into the observation files; summarize nothing there.

Complete the earlier lab’s cleanup first. Use fresh lab storage and the original three-service Compose file, so the topic and consumer groups below do not already exist. Create the observation and reassignment directories. Start the three nodes from kafka-labs/cluster/:

Wait until the quorum reports a leader:

Create the topic and describe it, saving the description as observations/00-topic.txt:

What you should see: six partition rows, each with three brokers in Replicas and the same three in Isr, and leaders spread across 1, 2, and 3. Write down which broker leads each partition; Phases 4 and 5 change that.

Run the producer benchmark in Terminal A. --throughput -1 removes the tool's rate limit, so the run measures how fast the producer and cluster can go with default batching:

The tool prints progress lines and then one summary line with records per second, MB per second, average and maximum latency, and the 50th, 95th, 99th, and 99.9th percentiles. Run it three times and save the full output, including batch-size-avg, record-queue-time-avg, and request-rate, for the Phase 2 baseline row. Then measure read throughput of the same records once with a fresh group:

Record in results.md:

Add the consumer result's MB.sec and nMsg.sec columns underneath.

What you should see: three summary lines that each report exactly 100000 records sent, and a spread between the three p99 values. Use the spread as an initial estimate of run-to-run variation. Repeat each variant before deciding whether a difference is meaningful.

Repeat the Phase 1 producer command, keeping --print-metrics and adding the properties shown for each variant; compare matched pairs that differ in one setting. The metrics dump lets you see the batching model directly: find batch-size-avg, record-queue-time-avg, and request-rate in the output.

Run these variants, one at a time:

Kafka 4.x uses linger.ms=5 by default, so the Phase 1 baseline already covers that setting. Compare variants 1 and 2 with the baseline for linger, variant 3 with the baseline for batch size, and variant 4 with variant 3 for compression. Repeat each variant three times and record the median of each metric, along with the run-to-run spread.

Tabulate in results.md:

For every row, write one sentence explaining the result with the batching model: work per record falls as records per batch rise, and a linger allowance only fills batches when records arrive fast enough to use it. The perf tool sends as fast as it can from one producer into six partitions, so state what that means for batch fill before you generalize anything. The payload is random letters, not order JSON, so the lz4 row measures the codec on this payload only.

What you should see: Test whether longer linger or larger buffers increase batch-size-avg. More records per batch can reduce requests per record, but request-rate may still rise if total throughput rises. Use queue time and batch size to test a possible explanation for the p99 change. Check broker latency, CPU, retries, and other signals if they do not explain it. If two rows differ by less than your Phase 1 spread, say so instead of ranking them.

Start a slow consumer in Terminal B. The pipeline inside the container sleeps after every record, so the consumer is the bottleneck by construction:

Because order-reporting has never committed, it starts at the end of the log and sees only new records. Start a rate-limited producer in Terminal A so arrivals are steady:

In Terminal D, sample the group six times, fifteen seconds apart, and save the output as observations/03-lag-samples.txt:

After the third sample, start a second instance with the exact Terminal B command in Terminal C. The group rebalances and each instance owns three partitions.

For every partition and sample, record CURRENT-OFFSET, LOG-END-OFFSET, and LAG, then compute two deltas between consecutive samples: the change in CURRENT-OFFSET and the change in LAG. Classify each interval as keeping up (LAG flat or falling), falling behind (CURRENT-OFFSET advancing but LAG rising), or showing no committed progress (CURRENT-OFFSET unchanged while LOG-END-OFFSET rises). Investigate repeated intervals with no progress before diagnosing a processing stall.

What you should see: in samples 1 to 3, inspect which partitions advance and which lag. Uneven routing, commit timing, and pipe buffering can make individual intervals differ. After the second instance joins, CONSUMER-ID differs between the two sets of three partitions and the extra processing capacity may flatten or reverse the LAG slope; record the actual trend. If an interval shows no committed-offset movement while the log grows, record it and investigate commit timing and processing before calling it a persistent stall. CURRENT-OFFSET moves only when the consumer auto-commits, about once per second, and the container's pipe buffer holds lines the sleep loop has not reached yet, so the committed position runs ahead of finished work.

Leave both consumer instances running. Start a longer rate-limited run in Terminal A with --num-records 60000 and --throughput 300, then about thirty seconds in, stop one broker from kafka-labs/cluster/:

In Terminal D, list under-replicated partitions and describe the group, saving both as observations/04-broker-stop.txt:

After a minute, restart the broker with docker compose start kafka-2. Rerun the under-replicated check every fifteen seconds until it prints nothing, and note the elapsed time. When the producer finishes, keep describing the group until every LAG is 0, and note how long that took.

Copy any Terminal A retry messages, including NOT_LEADER_OR_FOLLOWER or Connection to node 2, along with final errors and the summary. A run need not emit those exact warning strings.

What you should see: while kafka-2 is down, all six partitions appear in the under-replicated list with Isr missing 2, and the partitions it led show a new Leader. The producer may log retry warnings. All 60,000 sends can succeed if two replicas stay in sync and recovery completes within the delivery timeout; save final errors too if that condition does not hold. LAG may rise on partitions whose leader moved. Record which partitions change and whether lag drains after the run ends. After restart, 2 returns to every Isr, and it does not necessarily regain the leaderships it lost.

Add a fourth, broker-only node to kafka-labs/cluster/docker-compose.yml. It has no CONTROLLER listener and the voter list is unchanged, because it is not a voter:

Start only the new service with docker compose up -d kafka-4. Create reassignment/topics-to-move.json:

Copy it into a container and generate a plan for brokers 1,2,3,4:

If the tool reports that broker 4 does not exist, it has not registered yet; wait and retry. Save the two JSON documents it prints as reassignment/current.json and reassignment/proposed.json, review that every partition still lists three distinct brokers, and copy the proposal to kafka-1:/tmp/proposed.json. Start the Phase 4 producer command again in Terminal A so the move competes with live writes, then execute under a throttle. The value is 10,000,000 bytes per second, chosen to make the copy last long enough to watch; adjust it if the copy finishes before you can sample it:

Immediately verify, and repeat the verify and the topic describe every fifteen seconds until every partition reports completion. Save the outputs as observations/05-reassignment.txt:

When all partitions are complete, run the same command without --preserve-throttles, and inspect every affected broker’s replication rates and the topic’s throttled-replica lists to confirm cleanup:

What you should see: the execute command reports that reassignment started. The first verify may show work in progress, or the copy may already have finished. If you capture the intermediate state, describe shows 4 in Replicas and in Adding Replicas before it appears in Isr. For any partition whose current leader is not in its target list, Leader keeps the old broker until 4 joins Isr and that partition completes; only then does the leader change. The final verify prints lines about clearing broker-level and topic-level throttles, and no affected broker lists a replication throttled rate. The topic should also have no leader or follower throttled-replica list.

Run a long producer from a throwaway container on the compose network, so no restart kills it. Compose names the network after the project folder; confirm with docker network ls if the listing does not include cluster_default:

Now restart kafka-1, kafka-2, kafka-3, and kafka-4 in that order, one at a time, with docker compose stop followed by docker compose start. Before each stop, and again before the next, run the Phase 4 under-replicated check and the Setup quorum check until the first prints nothing and the second names a leader. While kafka-1 is offline, run those checks through docker exec kafka-4 with --bootstrap-server kafka-4:9092; while kafka-4 is stopped, use kafka-1. Save every check and the producer's warning and summary lines as observations/06-rolling-restart.txt.

What you should see: stops can produce under-replicated partitions and retry warnings; after each start, verify that replication recovers. At least two voters remain during each stop, so the quorum can elect a leader; brief checks may fail during an election. Save the final producer summary and any failed sends. All 120,000 sends can succeed if replication and recovery stay within the producer’s timeouts. If you had stopped a second node before the first rejoined Isr, affected writes could fail or retry until recovery, depending on replica placement, controller availability, and timeouts; the ISR check between steps is what prevents that.

Press Ctrl-C in Terminals B and C, wait for Terminal A to finish, then from kafka-labs/cluster/:

Verify that nothing remains:

The listing should contain no containers from this lab’s Compose project. Keep kafka-labs/tune-and-rebalance/ and the compose file with kafka-4; they are your submission.

You measured the same 100,000-record workload under one change at a time and explained each row with batch fill and request count instead of a rule of thumb. Repeated baseline and variant runs helped you estimate variation, and the producer's own batch-size-avg, record-queue-time-avg, and request-rate showed why a linger allowance helped or hurt at the rate this one producer could offer. The lag samples showed that LAG is a distance, not a rate: only the change in CURRENT-OFFSET between samples told you whether a partition was moving, and the pipe buffer showed that a committed offset can run ahead of finished work.

The broker-stop experiment connected the observed retries and send outcomes to ISR membership, recovery time, and producer timeouts. The reassignment checks tracked the new placement and replica recovery, including intermediate states when sampling caught them. For each rolling-restart step, stop one node, start it again, verify replica recovery and quorum health, then continue to the next node.