In this lab, you will run three instances of one Java consumer against a six-partition topic and record every revoke, assign, and lost callback with a timestamp. You will repeat the same join-and-leave sequence under the classic eager assignor, the classic cooperative assignor, and the consumer group protocol, then kill, freeze, and restart members to measure how long the group takes to notice and what static membership changes.

The shared setup stays the same across runs: one topic, orders.placed, with six partitions; one group, order-reporting; three members labelled A, B, and C; the same listener class; and UTC timestamps from one host clock. The phases reuse the same topic and listener. Each phase specifies which protocol, assignor, timing settings, identities, or failure actions to change. Record these along with the observed timing.

Use environment E1, the single broker kafka-local in Docker, and E3, the Java 21 Maven consumer project. The broker and every kafka-*.sh tool run inside the container through docker exec. The consumer JVMs run on the host, one terminal each, plus an admin terminal for describe commands and kill. Run the JVMs with java -cp rather than through Maven so that kill reaches the consumer process and not a Maven wrapper.

Start from the consumer project the consumers module set up and copy its pom.xml unchanged. Create:

Start from a fresh container so orders.placed does not already exist with another partition count:

Confirm with --describe --topic orders.placed that partitions 0 through 5 exist. The topic stays empty; a rebalance needs no records, and an empty poll result is a normal outcome.

Save this as src/main/java/com/example/kafka/RebalanceLab.java. The first argument is the member label, the second a properties file with the protocol and timing keys, and the optional third sets group.instance.id. Every callback prints one LAB | line: UTC timestamp, member, event, the partitions the callback receives, and the assignment the client reports at that moment.

The lab.revokeDelayMs system property adds a sleep inside the revoke callback. Leave it at zero except in Phase 6, where a slow revocation widens the window for sampling the group state.

Create config/eager.properties:

config/cooperative.properties:

config/consumer-protocol.properties holds only the protocol switch. The client's partition.assignment.strategy, session.timeout.ms, and heartbeat.interval.ms do not work under this protocol, so the file must not contain them:

config/fast-detection.properties shortens the classic liveness deadline. The heartbeat interval must stay below the session timeout, and the broker rejects a session timeout below its configured minimum; if a member fails to join with an error about the session timeout, raise the value and record what you used:

Build once:

Every phase starts members with this command, changing the label, properties file, and log name:

Extract a timeline with grep 'LAB |' logs/eager-A.log. The client prints its effective configuration at startup, so grep -m1 'session.timeout.ms' logs/eager-A.log shows the value in force for that run.

Start A with config/eager.properties and wait for ASSIGNED with owns=[0, 1, 2, 3, 4, 5]. The broker delays the first rebalance of an empty group by group.initial.rebalance.delay.ms, so exclude this initial delay from your later rebalance comparisons.

Start B. After both members print their final ASSIGNED callback and the group reports Stable, run:

Record each member's assignment. Start C, wait, and describe again. Then stop C with Ctrl-C and record what A and B print.

Fill observations/timeline-eager.md with one row per LAB | line: timestamp, member, event, callback partitions, and owns. Add the --members snapshots below the table.

What you should see. On every join, each existing member prints a REVOKED line whose callback set equals its whole assignment, then an ASSIGNED line with its new full set. Partitions that return to the same owner still pass through both callbacks. Because Ctrl-C makes a dynamic member leave the group during close, A and B react within seconds rather than after a session timeout. The --state describe shows range under ASSIGNMENT-STRATEGY.

Stop all members. Repeat the sequence with config/cooperative.properties and log names cooperative-*.log: start A, B, C, describe after each join, then stop C.

Fill observations/timeline-cooperative.md in the same format, adding a column that counts ASSIGNED lines per member per join.

What you should see. When B joins, A prints one REVOKED line naming only the partitions that move, never one naming partitions it keeps. B prints an ASSIGNED line with an empty callback set, then a second carrying the moved partitions: two rounds, visible as two callbacks. When C joins, A and B each release a subset. When C leaves, its partitions have no owner, so A and B print ASSIGNED lines with no REVOKED line first. ASSIGNMENT-STRATEGY reads cooperative-sticky.

Stop all members. Repeat with config/consumer-protocol.properties and log names consumer-*.log. After C joins, run the --members --verbose describe and:

Record every column. Kafka 4.3's --list command also accepts a --type filter separating classic from consumer groups; check kafka-consumer-groups.sh --help for the spelling and confirm which type order-reporting is now.

Fill observations/timeline-consumer.md. Below the table, list the keys that left the configuration since Phase 2 and the broker setting that now governs the session timeout, as named in the consumer-protocol chapter's configuration table.

What you should see. Members still print REVOKED only for partitions that move and ASSIGNED for partitions they gain, but no group-wide round shows in the callbacks: A releasing a partition and C gaining it are individual heartbeat exchanges, and B's transfer can interleave with them. ASSIGNMENT-STRATEGY no longer shows a class-derived name; it shows the server-side assignor, uniform unless you set group.remote.assignor. The --members --verbose output may carry extra columns for this group type; record whatever it prints.

This phase measures the interval from a member going silent to the survivors receiving its partitions: t0 is your date output, t1 the first survivor's ASSIGNED line. It kills and freezes consumer JVMs only, never the broker. The pgrep -f pattern matches on the member label, so keep labels unique.

Default classic timing. Stop all Phase 3 members and verify no RebalanceLab process remains. Then start A, B, and C with config/cooperative.properties and log names detect-default-*.log. Once settled, record t0 and kill C in one line:

Record t1, the difference, and the session.timeout.ms value from detect-default-A.log.

Lowered classic timing. Stop A and B. Start all three with config/fast-detection.properties and log names detect-fast-*.log, settle, and repeat the kill. Record t0, t1, the difference, and the session and heartbeat values from the log.

Frozen member. Stop A and B. Start all three with config/cooperative.properties and log names detect-freeze-*.log. Freeze C so the process exists but nothing in it runs:

After A and B print ASSIGNED, resume C and record what it prints:

Consumer protocol timing. Stop everything. Start all three with config/consumer-protocol.properties and log names detect-consumer-*.log, settle, and repeat the kill -9. There is no client-side session timeout to lower here; record the detection time against the server default named in the consumer-protocol chapter.

Fill observations/detection-times.md with one row per run: protocol, session and heartbeat settings in force, action, t0, t1, difference, and the survivors' callbacks. For the freeze run, add C's callbacks after kill -CONT.

What you should see. In the default run, A and B stay quiet for a period close to the session timeout in the log, then print ASSIGNED for C's partitions with no REVOKED before it. In the lowered run, the same evidence arrives after a much shorter gap. The shorter heartbeat interval increases request traffic; the shorter session timeout reduces tolerance for pauses. A frozen process looks like a crashed one from the coordinator's side; the difference appears after kill -CONT, when C prints LOST for the partitions it believed it owned and then rejoins, causing one more rebalance on A and B. The consumer-protocol run detects the kill on the server's schedule, and the client log shows no session.timeout.ms you could have changed.

Stop all members. Start A, B, and C with config/cooperative.properties, log names static-*.log, and instance ids reporting-0, reporting-1, and reporting-2 as the third argument. Settle and record the assignment with --members --verbose.

Restart inside the window. Stop B with Ctrl-C. While it is down, run the --state describe and record #MEMBERS. Restart B immediately with reporting-1, well inside the session timeout you read in Phase 4.

Restart after the window. Stop B with Ctrl-C again and wait at least 60 seconds, longer than the default session timeout. Watch A and C. Then restart B with reporting-1 and watch all three.

Duplicate identity. With all three running, start a fourth process labelled D with reporting-1, the id B holds, logging to static-D.log. Record which process prints an ERROR line and the exception name.

Duplicate identity under the consumer protocol. Stop everything. Start A, B, and C with config/consumer-protocol.properties and the same instance ids, settle, then start D with reporting-1. Record the exception name.

Fill observations/static-membership.md with the four scenarios, each member's callbacks, and the #MEMBERS value while B was down.

What you should see. After the first stop, #MEMBERS still counts B: the default close retains a static member's membership. When B returns in time, A and C print nothing and B prints ASSIGNED with the partitions it owned before. After the 60-second absence, A and C print callbacks when the coordinator expires reporting-1, and B's return causes a second set. In the classic duplicate run, one of the two processes sharing reporting-1 prints FencedInstanceIdException; do not assume in advance which. Under the consumer protocol, the coordinator rejects the new process, and its client reports UnreleasedInstanceIdException and B keeps its partitions.

Stop all members. Start A and B with config/eager.properties, adding -Dlab.revokeDelayMs=4000 before -cp so each revocation takes four seconds. Start this sampler in the admin terminal and, while it runs, start C:

Record the COORDINATOR (ID) column and every distinct STATE value, with the time of the first sample showing each. Stop all three members and verify that no RebalanceLab process remains. Restart A and B with config/consumer-protocol.properties, wait for their assignments to settle, then repeat the sampler while starting C with that same file.

What you should see. The coordinator id matches the single broker's node id, the leader of the __consumer_offsets partition this group hashes to. During the classic join, samples may capture PreparingRebalance or CompletingRebalance before Stable. Under the consumer protocol you may observe Reconciling; polling can miss this state and Assigning. Each sample takes longer than a second because docker exec has its own startup cost, so treat a missing intermediate state as a sampling gap, not proof the group skipped it.

Stop every consumer with Ctrl-C, then confirm none remain:

Verify with docker ps -a --filter name=kafka-local, which should print only the header. Keep logs/, observations/, and report.md.

The three timelines show how each approach handles the same membership change. The eager assignor revoked every partition on every join, the cooperative assignor revoked only the partitions that moved and needed a second coordination round, and the consumer protocol replaced group-wide rounds with per-member heartbeat exchanges in which the coordinator's server-side assignor sets the target and each member reconciles toward it.

The detection-time table separates what the protocol decides from what a timeout decides. The coordinator detects a killed or frozen member when its session expires, and shortening that window allows faster takeover but reduces tolerance for pauses. Shortening the heartbeat interval separately increases heartbeat traffic. Static membership shifts the trade-off again: a member that returns inside the window keeps its partitions and disturbs nobody, one that returns late triggers the reassignment anyway, and two processes cannot share one static identity: the classic protocol may fence a process, and the consumer protocol may reject the new claimant.