A consumer application needs to do more than retrieve records. It needs a stable identity, a clear definition of completed work, and a way to stop without saving progress past unfinished records.

We’ll build a small Java application that reads order events, prints their keys and values alongside their Kafka positions, and saves progress after processing each batch. Then we’ll restart it and check where it resumes. The example keeps processing simple so the consumer’s behavior remains easy to see.

Use Java 21, Maven 3.9 or later, and Docker with Linux container support. The application uses kafka-clients version 4.3.1 and connects to a local Kafka broker. Run the shell commands in Bash, Zsh, or a Bash-compatible WSL terminal.

Check the tools on your host:

Maven must report Java 21 or later. Docker must report a running server as well as a client. The first build needs internet access to download dependencies.

We’ll use a container named kafka-local, running apache/kafka:4.3.1, with Docker publishing broker port 9092 at 127.0.0.1:9092. If this learning container already exists and is stopped, start it with docker start kafka-local. If it is already running, leave it running.

If you do not have that container, create it with:

This image starts a combined broker and KRaft controller using its default configuration. The single node has no backup replica, and connections use plaintext without authentication. Keep this setup local and use synthetic data.

Wait until Kafka can list topics successfully:

The Java application runs on the host. Kafka’s console producer runs inside the container. Both can use localhost:9092 with this particular port mapping and the image’s default advertised address.

The diagram shows the two clients and where our application performs its work:

Printing is the complete processing action for this example. It is not a durable reporting system, and committing an offset does not make terminal output durable.

Use a dedicated topic so existing order events do not affect the walkthrough:

One partition gives the sample records a single offset sequence. One replica matches the single available broker.

If the topic already exists, choose a fresh topic name and replace it in the commands and Java constant below. Also choose a fresh group name in the Java configuration and group-inspection command. Keep other producers and consumers away from these example names while checking the expected results.

Verify the topic:

The description should show partition 0, a replication factor of 1, and an assigned leader. Keep Kafka running while you build the application.

Create a directory for a small Maven project:

Save the following as pom.xml in this directory. Maven uses it to select the Java language level, resolve the Kafka client and logging dependencies, and prepare the runtime libraries.

slf4j-simple makes the client’s logs visible, including connection warnings. Its version matches the logging API this Kafka client uses. You do not need an application framework for the example.

Save this complete class as src/main/java/com/example/kafka/OrderConsumer.java.

One thread owns the consumer, processes records sequentially, and commits after the entire returned batch succeeds. A shutdown hook, which Java invokes during an orderly JVM shutdown, signals that thread to stop and waits briefly for cleanup.

The generic types KafkaConsumer<String, String> describe the deserialized key and value. StringDeserializer decodes their bytes into strings. The JSON value remains a string; this program does not parse its fields or validate the order schema.

The handler checks for a nonempty key and value because that is the contract for this example’s order events. A null value is valid in Kafka and has a specific use in compacted topics, but this application rejects it. An application’s event contract can be narrower than Kafka’s record format.

System.out.checkError() catches output failures that Java’s PrintStream reports through its error flag rather than throwing. Even with this check, terminal output is only a local demonstration of processing.

Configuration Choices

The settings make the application’s identity and recovery behavior explicit:

The one-second poll() duration lets the call wait for data. It does not require every call to take a second or limit processing to one second. The five-second commit and close timeouts are also local choices, not general recommendations for a deployed service.

Calling subscribe() registers the topic interest. The polling loop then lets the client join the group, obtain an assignment, and return records. The startup message confirms that the application reached the loop; it does not confirm a successful broker connection or assignment.

For each nonempty result, the code runs every record through process(). Only after all calls return successfully does it commit records.nextOffsets(). This map contains the next positions for the returned partitions, including the offset metadata the client supplies.

The diagram shows where success and failure lead:

If processing throws halfway through a batch, execution never reaches the commit. The outer error handler reports the failure and lets the exception terminate the application. Closing the consumer does not automatically commit here because we disabled automatic commits.

This is a deliberate policy for a small application: stop so the failure is visible and recover from saved progress. Catching an error, printing it, and continuing to commit the whole batch would risk skipping the failed record on restart.

There is no commit in finally. That block runs after failures as well as successful work, so an unconditional commit there could save progress beyond unfinished records.

The example still permits duplicates. If printing succeeds and the process exits before a commit succeeds, those records can print again. A commit timeout can also leave the outcome uncertain. Replacing printing with a database write requires a way to tolerate repeated events, such as applying a unique event ID and the business update in one database transaction.

Open two terminals. Use Terminal A for the Java consumer and Terminal B for Kafka commands.

In Terminal A, build from the directory containing pom.xml:

Maven places compiled classes in target/classes and dependency JARs in target/lib. After the build succeeds, launch a separate Java process:

The classpath combines the compiled application with its runtime libraries. The quotes leave the JAR wildcard for Java to interpret. This command uses the classpath separator for macOS, Linux, and WSL.

Keep the process running. Initially, it should wait because the fresh topic is empty. If it repeatedly logs connection failures, fix those before publishing.

In Terminal B, publish three synthetic orders:

Each line uses the text before | as the Kafka key and the JSON after it as the value. The quoted here-document passes these lines without shell expansion. Keep the closing EOF on its own line. These records have no custom headers.

--sync makes the producer wait for each send result. With one replica, acks=all still provides only one stored copy. A successful producer exit says nothing about consumer processing.

For a fresh topic with no other writes, Terminal A should print these record lines, possibly with log messages and commit messages between them:

The three records need not arrive in one poll. Timing determines the batch boundaries. After successful processing and commits, the saved offset for partition 0 should reach 3.

Do not rerun the producer merely because it finishes quietly. A second invocation can append another three records with the same event IDs at new offsets.

In Terminal B, inspect the group while the consumer remains running:

After the consumer processes all three records and commits their next offset, expect this state for partition 0:

If the offset is lower, check the application for processing or commit errors. The command may also run before the consumer’s latest commit finishes.

Once the committed offset reaches 3, press Ctrl-C in Terminal A. Wait for the Java process to exit, then run the same Java command again.

The consumer should wait for new records without reprinting offsets 0 through 2, provided the group’s saved offset remains valid. auto.offset.reset=earliest is a fallback for a missing or invalid offset; it does not force a replay on every start.

Publish one additional event in Terminal B:

Under the same fresh-topic assumptions, the consumer prints this event at offset 3 and commits 4. This confirms that it resumed from saved progress and continued reading live data.

Changing client.id would not reset that progress. Changing group.id would select a different group, which could read the retained events from the beginning under this starting-position policy.

The Java consumer is not thread-safe, so the shutdown hook does not call close() or commit offsets. It sets a stop flag and calls wakeup(), which is specifically safe to call from another thread. The main thread catches the resulting WakeupException and performs cleanup.

The hook waits on closed so the JVM gives the main thread an opportunity to close the consumer before shutdown completes. The hook waits at most ten seconds in this local example. A forced process kill bypasses this cleanup.

If Ctrl-C arrives while the application is printing a batch, it can finish that processing before reaching another consumer operation. A pending wakeup may interrupt the commit, leaving the batch eligible for replay. The program allows completed work to replay if its offset commit fails. It never commits unfinished work.

Real handlers need their own deadlines. wakeup() interrupts supported consumer operations; it does not cancel a blocked database request or arbitrary application code. A production shutdown budget must account for those operations as well as consumer cleanup.

Use the following checks when the walkthrough behaves unexpectedly:

The client handles eligible transient network failures internally. This program exits when an error escapes into the application, including a failed commit, rather than adding an unbounded retry loop. After resolving the cause, restart with the same group to recover from its saved position.

For a permanently invalid event, a deployed service needs an explicit handling policy. Blind restarts can repeatedly stop at that record; blindly skipping it can omit required work. This small program makes the failure visible and leaves that decision to the operator.

When finished, press Ctrl-C to stop the consumer. Kafka and the topic can remain available for further local work.

You’ve built a Java consumer with an explicit group identity, string deserializers, sequential processing, manual batch commits, and coordinated shutdown. You’ve also published keyed events and checked that a restart resumes from the group’s saved offset.

The code saves progress only after every record in a returned batch succeeds. Failures can still repeat completed work, so any durable business action you add to the handler needs to tolerate replay.