In this lab, you publish order events to one broker and test four behaviors: records remain after reads; groups track progress independently; a restarted consumer resumes from its committed offset; and a new group can replay records Kafka still retains. You use only the console tools bundled in the apache/kafka:4.3.1 image, so you do not write client code.

Use the following topic and event batches. The topic is orders.placed with three partitions. Batch 1 holds six OrderPlaced events for orders ord-1042 through ord-1047, batch 2 holds three more for ord-1048 through ord-1050, and you publish each batch exactly once. With nine records in the topic, every LOG-END-OFFSET total, every lag value, and every replay count has one correct answer you can check your observations against.

Environment: E1, the single combined broker and controller in a container named kafka-local. Docker runs on your host. Every Kafka tool runs inside the container through docker exec; the only host files are the event batches and your observations. Open three Bash-compatible terminals: Terminal A for the order-reporting consumer, Terminal B for the order-fulfillment consumer, and Terminal C for everything else.

Create:

Run every command from kafka-labs/order-pipeline/ so the producer's relative file paths resolve.

Each line in a batch file is a key, a pipe character, and a JSON value, in the format the console producer accepts when you enable key parsing. Create events/batch-1.txt:

Create events/batch-2.txt:

Keep each event on one line with no blank lines; the producer treats every line as one record.

If a kafka-local container already exists from the chapters, remove it first so the lab starts from a known state: docker stop --time 30 kafka-local followed by docker rm -v kafka-local. Then start a fresh broker in Terminal C:

Confirm the container is running and check the tail of its logs:

Now ask Kafka itself. A topic listing that returns with exit status 0 proves the broker answers requests, and the quorum check proves the controller side is up:

Record both outputs in observations/01-readiness.txt.

What you should see: the container status is Up, the topic listing prints nothing because no topics exist yet, and the quorum output names node 1 as the leader with a cluster ID. If the listing fails with a connection error, the broker may still be starting; wait and retry.

Create orders.placed with three partitions. The replication factor must be 1 because there is one broker:

Describe it and save the output as observations/02-topic.txt:

Publish batch 1. The -i flag keeps standard input open so the producer can read the file, parse.key=true splits each line at the first pipe, and --sync with acks=all waits for the broker to acknowledge each write before sending the next:

The producer exits quietly on success. Do not run it again.

What you should see: the describe output shows PartitionCount: 3, ReplicationFactor: 1, and three partition rows numbered 0, 1, and 2, each with Leader: 1, Replicas: 1, and Isr: 1. No consumer is running yet, so nothing prints the records. Kafka has already stored them, and the next phase proves it.

Start the reporting consumer in Terminal A:

Start the fulfillment consumer in Terminal B with the same command, changing only the group to order-fulfillment. Leave both running.

Each group has one member, so that member receives all three partition assignments. Kafka tracks a separate committed offset for each group and partition; watch for that separation in the group descriptions.

Once both terminals have printed records, describe both groups from Terminal C and save the output as observations/03-two-groups.txt:

Also write down the partition and offset Terminal A printed for each key, and confirm Terminal B printed the same values.

What you should see: both terminals print the same six records at the same partitions and offsets, because they read the same stored log. Each group description has three rows, one per partition. In every row CURRENT-OFFSET equals LOG-END-OFFSET and LAG is 0, and the three LOG-END-OFFSET values add up to 6. The split across partitions depends on how the keys hash; record it rather than predicting it. If a CURRENT-OFFSET still trails, wait for the next auto-commit and describe again.

Press Ctrl-C in Terminal A to stop the order-reporting consumer. Leave Terminal B running. Describe order-reporting from Terminal C; the committed offsets survive the stop.

Publish batch 2:

Wait until Terminal B has printed all three new records and its next auto-commit makes order-fulfillment’s lag zero. Then describe both groups and save both outputs as observations/04-reporting-stopped.txt.

What you should see: Terminal B prints the three new records as they arrive. The order-reporting description reports that the group has no active members, its CURRENT-OFFSET values are unchanged from Phase 3 and still add up to 6, its LOG-END-OFFSET values add up to 9, and its LAG values add up to 3. The order-fulfillment description shows LOG-END-OFFSET totaling 9 with every LAG back at 0. Stopping reporting did not stop fulfillment’s processing.

Now restart the consumer in Terminal A with the exact Phase 3 command, including --from-beginning. Wait until the next auto-commit makes reporting’s lag zero, describe both groups again, and save both outputs as observations/04-reporting-resumed.txt.

What you should see: Terminal A prints only the batch-2 keys ord-1048, ord-1049, and ord-1050. The six batch-1 records do not print again, even though the command says --from-beginning, because the group has a valid committed offset for every partition and that offset wins. After the commit, order-reporting shows LAG of 0 on all three rows and CURRENT-OFFSET totaling 9.

Neither running group can show the whole history again without moving its position. A brand-new group can. From Terminal C, run a consumer in the group first-orders-reader, which has never committed anything, and let it exit after nine records:

Save the nine printed lines as observations/05-replay.txt, then describe all three groups and append those outputs. Finally, look at where the broker keeps these records on disk:

What you should see: the replay prints nine records, one per event ID from evt-7001 to evt-7009, at the same partitions and offsets Terminals A and B reported earlier. Records from different partitions may interleave; within one partition, offsets ascend. The order-reporting and order-fulfillment descriptions are unchanged by the replay. The log directory listing contains orders.placed-0, orders.placed-1, and orders.placed-2, one directory per partition, alongside a set of __consumer_offsets- directories where the broker stores the committed offsets of all three groups. No producer republished the records, and reading the records a third time did not remove them.

Press Ctrl-C in Terminals A and B. Then stop and remove the container:

Start a fresh broker with the Phase 1 docker run command, wait until the topic listing succeeds, then list topics, list consumer groups, and list the log directory. Save all three outputs as observations/06-reset.txt:

What you should see: the topic listing is empty, the group listing is empty, and the log directory has no orders.placed- directories. Describing orders.placed fails with an error saying the topic does not exist. The records lived in the container's filesystem under /tmp/kraft-combined-logs, and docker rm -v deleted that filesystem. A stop and start would have kept everything; removal did not.

Stop any running console consumers with Ctrl-C, then remove the broker:

Verify that nothing remains:

The listing should show no container. The image stays downloaded for later modules. Keep the kafka-labs/order-pipeline/ folder; it is your submission.

You published nine keyed records once and read them through two live groups, a restarted consumer, and a replay group that started from offset 0. The group descriptions gave you the numbers behind the module's claims. LOG-END-OFFSET measured the log, CURRENT-OFFSET measured one group's committed position, and LAG was the gap between them. Publishing batch 2 increased order-reporting's lag while it was stopped. order-fulfillment kept reading and returned to zero lag.

The restart showed that a valid committed offset takes precedence over --from-beginning, and the replay showed that reading never removes anything. Kafka stored the records in files under /tmp/kraft-combined-logs, next to the __consumer_offsets partitions holding each group's progress, and they lasted exactly as long as the container that held them. A work queue would have handed each batch-1 event to one of the two workers and forgotten it on acknowledgment; Kafka kept the log and let every group decide where to stand in it.