Kafka becomes easier to understand when you publish a few records and watch them arrive at a consumer. Seeing each record’s key, value, partition, and offset helps connect the concepts to what actually happens.

In this chapter, we’ll use Kafka’s built-in console tools to publish sample order events, consume them in real time, check how far a consumer has read, and replay stored records. We’ll work with synthetic data and a single-partition topic so the flow is easy to follow.

Start with a running container named kafka-local using the apache/kafka:4.3.1 image and its default combined broker and KRaft controller configuration. The broker should be reachable at localhost:9092 inside the container.

This setup uses a single node and plaintext connections. Use it for learning, not production.

Open two Bash-compatible terminals. We’ll call them Terminal A and Terminal B. Run all commands on your host machine. Each docker exec command runs Kafka’s tools inside the container, so you don’t need to install Java on your host.

In Terminal B, check that the broker is ready:

Continue once the command succeeds. If you see a connection error, resolve it before moving on.

Next, create a topic for this walkthrough:

Use a fresh topic so earlier records don’t affect the examples. If orders.placed.first-run already exists, choose another name and use it in every command that follows. Keep other producers from writing to this topic, and make sure its records don’t expire during the walkthrough.

Check the topic’s layout:

The output should show one partition, numbered 0, a replication factor of 1, and an assigned leader. Since the topic has only one partition, every record we publish will go to partition 0.

In Terminal A, start a consumer before publishing any events:

The consumer will keep running and wait for records. If the topic is empty, you won’t see any output yet. Leave it running while you work in Terminal B.

Here’s what the options do:

The diagram shows how records move through the example. The consumer asks Kafka for records, and Kafka returns them in fetch responses:

These console tools are real Kafka clients. The producer sends records to the broker, and the consumer reads them over Kafka’s network protocol. In this example, the consumer simply prints each record to the terminal.

Each input line contains a key, a pipe character (|), and a JSON value. The key, such as ord-1042, identifies the order. The eventId inside the JSON, such as evt-7001, identifies a specific event for that order.

Run this complete command in Terminal B:

The <<'EOF' block is a here-document. It passes the three event lines to the producer as input. Quoting 'EOF' keeps the shell from expanding variables or running command substitutions in the data. Keep each event on one line and the closing EOF on its own line. That closing marker ends the input; the producer does not send it to Kafka.

The -i flag keeps Docker’s standard input open so the producer can read these lines. We don’t need -t because the input comes from the block above.

With parse.key=true, the console producer splits each line at the first |. Everything before it becomes the key, and everything after it becomes the value. The console tool discards the pipe itself. This parsing happens in the console tool.

We’re using JSON for the values, but the console producer does not check the order schema or interpret the fields. Here, totalMinorUnits is an amount in paise, so 129900 means INR 1,299.00. Kafka stores both the key and value as bytes.

The --sync option makes the producer wait for each send to succeed or fail before sending the next record. This makes the walkthrough easier to follow, though it limits throughput. The acks=all setting asks all current in-sync replicas to acknowledge the write. Our topic has only one replica, so there is no backup copy if the broker loses its data.

The producer may finish without printing a success message. Check for errors, then look at Terminal A for the records. Don’t rerun the command just because it finishes quietly: running it again can add three more records with the same event IDs.

If you started with a fresh topic and published the events once, Terminal A should show these three records. You may also see log messages.

Notice a few things:

The diagram shows the stored records and the position where Kafka would append the next record:

With no further writes, the log end offset is 3. Once a consumer has finished processing the record at offset 2 and all earlier records, it can save 3 as the position to resume from. This points to the next record to read; it does not mean a record already exists at offset 3.

The records appear in this order because one producer sent them sequentially to one partition. This example does not show ordering across partitions. An application that processes records in parallel could also finish its work in a different order.

Leave the consumer running in Terminal A so it can keep polling Kafka and saving its progress. In Terminal B, inspect the consumer group:

Once the consumer commits its offset, the row for orders.placed.first-run, partition 0, should show:

Consumer IDs and host details will vary. If the committed offset is missing or below 3, leave the consumer running and check again after it has had time to poll and commit.

Zero lag means the group’s saved progress has caught up with the log. It does not tell you whether a database update, payment, or email has completed.

After you see CURRENT-OFFSET reach 3, press Ctrl-C in Terminal A to stop the consumer. Then run the same consumer command again. As long as the saved offset and topic data still exist, the consumer should wait for new records without printing offsets 0 through 2 again. The saved offset takes precedence over --from-beginning.

Automatic commits make this walkthrough easy to observe. In a real application, you need to decide when processing is complete and when it is safe to save progress. If the application crashes after performing an action but before committing, it may repeat that action after restarting. If it commits too early, it may skip unfinished work.

To read the original records again without changing the saved progress of first-orders-reader, run a separate consumer in Terminal B:

This consumer reads partition 0 directly, starting at offset 0. It does not use the live consumer’s group identity and has automatic commits disabled, so it leaves first-orders-reader’s saved position unchanged.

You should see the same three records, with their original offsets. The consumer exits after reading the third record. If it waits 30 seconds without receiving another record, it times out. A timeout does not mean the consumer read all three records. Check the output.

This is replay: reading stored records again. No producer republishes the records, and Kafka assigns no new offsets.

Starting another consumer with --group first-orders-reader would behave differently. It would join the existing group and share the partition assignments. Since this topic has only one partition, only one member would read from it once the assignment settles.

Replay only works while the records are still available. If Kafka has removed them through retention, this command cannot bring them back.

Reading an event again can also repeat whatever work your application performs. For example, an application that creates invoices should recognize an event it has already handled and avoid creating a duplicate invoice.

Most issues in this walkthrough come from input formatting, saved consumer progress, or the local Kafka setup.

If you see the same event ID at different offsets, those are separate records in the topic. Kafka does not automatically remove duplicates based on the JSON eventId. Producer retry protections and application duplicate handling address different causes of repeated records.

When you’re done, press Ctrl-C to stop any running console consumers. The here-document producer and the replay consumer exit on their own. Stopping these tools leaves Kafka running and keeps the topic intact.

You can leave the broker running for more practice or stop it while keeping the container:

Your topic and saved consumer progress remain available in the same container. When you return, you can resume from the saved position or replay earlier records.

You’ve published records with separate keys and JSON values, inspected their partitions and offsets, and checked a consumer group’s saved progress. You’ve also replayed stored records without changing where that group resumes.

Publishing a record, printing it, and saving consumer progress each confirm a different step. This walkthrough shows how records move through Kafka. A real application also needs to recover from failures and avoid repeating business actions when it reads an event again.