An online store creates an invoice for every accepted order. If the invoicing database is temporarily unavailable, the consumer must keep the order eligible for another attempt. Moving on and forgetting it would leave an order without an invoice.

At-least-once processing preserves that unfinished work. The consumer completes the required operation before saving its Kafka progress. If recovery cannot tell whether the operation finished, it tries again, which means the application may repeat successful work.

In this chapter, we’ll follow an order through processing, offset commits, failures, and recovery. We’ll build a Java consumer loop, examine how batches and rebalances affect repeated work, and define the conditions needed to keep this guarantee meaningful.

At-least-once processing means that records remain eligible for processing until the application completes the required work. Failures may cause multiple attempts, including attempts that repeat an already successful operation.

This is a conditional guarantee. The input must remain available, the consumer must recover, and the operation must eventually be able to succeed. Repeatedly calling a broken handler does not guarantee a useful result. The recovery policy must preserve the work while operators resolve the underlying problem.

We’ll use an ordinary consumer group named order-invoicing in a KRaft-based Kafka cluster. Assume the group has a valid committed position of 42 for partition 0 of orders.placed. The next record has offset 42, key ord-1042, and this value:

The order amount is ₹125.00, or 12500 paise. The consumer’s handler, the application code responsible for a record, creates the invoice in a database.

For this example, processing succeeds only when the database transaction that creates the invoice has committed, or the handler has verified that the intended invoice already exists. Putting the request in an in-memory queue, starting an asynchronous call, or executing an insert inside an uncommitted transaction does not meet that definition.

This definition gives the offset a concrete meaning: advancing beyond an order says the application has created or verified its invoice.

For offset 42, the consumer reads the record, completes invoice creation, and then commits 43 as the next position for partition 0.

The diagram shows the success path. The database commit and Kafka offset commit are separate operations.

If processing fails before the invoice exists, the consumer does not advance its saved position past 42. A replacement can resume there and attempt the work again, provided the record is still available.

An offset commit is a position for an entire partition, not an acknowledgment attached only to one record. Committing 43 tells recovery to skip every earlier position in that partition. All required work before that position must therefore be complete.

The core rule is: never commit past unfinished work. It applies whether the consumer handles one record at a time, processes a batch, or dispatches records to worker threads.

Suppose the database commits the invoice for ord-1042, then the consumer crashes before Kafka saves offset 43. The group’s committed position remains 42.

The replacement consumer has no Kafka progress record telling it that invoicing succeeded. It reads offset 42 and invokes the handler again.

Kafka may contain only one copy of evt-7001. The repetition comes from recovery processing that same record again. If the handler blindly inserts a new invoice on every invocation, the customer can end up with two invoices.

The interval between the completed database work and the saved Kafka position is the duplicate processing window. Committing immediately after each record makes that window smaller, but cannot remove it while the two commits remain independent.

Different failure points produce different recovery outcomes:

A commit timeout does not prove that Kafka rejected the commit. This uncertainty is acceptable for at-least-once processing because the work was already complete before the commit request. Recovery can either skip completed work or repeat it; neither outcome requires skipping unfinished work.

The following helper uses Java 21 and org.apache.kafka:kafka-clients:4.3.1. It runs inside an existing application. The caller provides connection properties and a synchronous invoice handler; the database implementation is outside this example.

Use a reachable KRaft-based Kafka cluster where you have already created orders.placed. Supply bootstrap.servers, group.id=order-invoicing, and the authentication and encryption properties that cluster requires. A local learning broker may use localhost:9092 if that matches its advertised listener.

The helper uses auto.offset.reset=none, so each assigned partition needs a valid committed starting position. If a position is missing or no longer usable, the application fails instead of silently choosing another start. Initial group setup must deliberately select the first records the application is responsible for processing.

The handler must return only after invoice creation is durable or the handler verifies the intended invoice. It must throw if the work fails or remains uncertain. An application that starts background work and immediately returns does not satisfy this contract.

The code processes all returned records before attempting a batch commit. records.nextOffsets() supplies the next positions for the returned partitions, including available leader-epoch metadata. For partition 0 containing offsets 42 through 46, the next position is 47. Using the returned metadata also avoids assuming every offset has a visible record.

A handler exception exits the loop before the batch commit. Closing the consumer does not automatically commit this unfinished batch because the code disables automatic commits. The consumer can process successful records earlier in the batch again after restart, while the failed record remains eligible for processing.

A commit exception also exits the method. The batch’s work has already succeeded, so recovery may repeat completed operations. This helper chooses a simple recovery policy: let the application report the failure, then restart with a fresh consumer using the same group. The surrounding service must arrange that restart, with a delay between attempts, or request intervention when the error needs a fix. Exiting alone preserves the position but does not complete recovery.

enable.auto.commit=false gives this code control over saving progress. It does not establish at-least-once processing by itself. The handler’s completion contract, the placement of the commit, and the failure path provide the behavior.

Automatic commits can also support at-least-once handling in a correctly structured synchronous loop. They become unsafe when polling or closing can save progress before returned records finish. Explicit commits make the required boundary visible here.

A subtle mistake is to catch a handler failure, omit the commit, and simply call poll again. That does not make Kafka immediately return the failed record.

The consumer has two relevant positions. Its current position advances as poll returns records. Its committed position is the saved recovery point. Omitting a commit leaves the recovery point unchanged, but does not rewind the current position.

Consider this intentionally flawed recovery path. Assume partition 0 contains consecutive offsets and the example includes no other partition:

The later commit passes the failed record and the unattempted remainder of its batch. The application has lost work despite “not committing on failure.”

The Java helper avoids this by exiting on the first failure. A fresh consumer returns to the group’s committed position. Another valid design can retain and retry the failed work before advancing, or explicitly reposition the affected partition to the earliest unfinished record while it still owns that partition. Either approach needs to account for the rest of the batch as well.

Retries need time to be useful. When the database is unavailable, immediate repeated calls can increase load without improving the chance of success. A delay between attempts, often called backoff, gives the dependency time to recover. Requests should have bounded timeouts so an attempt cannot hang indefinitely.

A poison record repeatedly fails because of its contents, such as an unsupported event version. Waiting longer will not fix it. The application needs a schema or code fix, corrected data, or a durable route for investigation and eventual reprocessing.

A dead-letter topic stores records the application could not handle normally. Moving an order there does not mean its invoice exists. If the business requirement remains “every order gets an invoice,” the recovery workflow must still finish that work. The destination must confirm a durable handoff before the source offset advances; otherwise a crash can lose the only actionable copy. Repeating a handoff can also create duplicates, so its recovery policy matters too.

Batch commits trade fewer commit requests for more possible replay. If the helper completes offsets 42 through 46 and crashes before committing 47, all five can run again. If it fails at 44, successful work at 42 and 43 can repeat alongside the retry of 44.

Smaller batches or more frequent commits can reduce repeated work. They add commit overhead and do not eliminate the duplicate processing window. Choose the frequency based on processing cost, acceptable replay, and measured throughput rather than assuming every record needs its own commit.

Completion Within a Partition

Parallel processing needs to track gaps. Suppose workers handle three records from partition 0, and the saved position is initially 42:

Offsets summarize progress through a partition. They cannot express “skip 44 because it is done, but retry 43.” Recovery from 43 can therefore repeat 44 even though its worker already succeeded.

Completion tracking is separate for each partition. A gap in partition 0 does not inherently prevent saving completed progress in partition 1. The example helper commits only after the whole polled batch succeeds for simplicity; a more selective implementation can commit each partition’s safe position independently.

Ownership Changes

A rebalance can cause replay without a process crash. Suppose a consumer finishes an invoice but loses its partition before saving progress. Its replacement can resume from an earlier committed offset and attempt that invoice again.

Losing ownership does not cancel a database request already in progress. The old request and the replacement’s attempt may overlap. Destination-side protection must handle concurrent duplicates as well as sequential retries.

Long processing also affects group membership. The Java consumer limits the allowed time between polls through max.poll.interval.ms. If handling and retries take too long, the group can reassign partitions, with timing depending on the group configuration. max.poll.records can limit how many records one poll returns, but an unbounded handler still defeats any batch-size choice.

Applications that use workers must keep consumer API calls on the consumer thread, manage polling and backpressure, and save only known completed progress. During shutdown, drain work within a bounded time or leave it uncommitted for recovery. An unconditional final commit of fetched positions can skip unfinished records.

At-least-once processing is useful when an operation must survive temporary failures and the destination can tolerate or reject repeated attempts. Invoice creation needs both parts: preserving the order until the application handles it and ensuring retries do not create extra invoices.

For this example, assume the business allows one invoice per order. The database can enforce uniqueness on orderId, and the handler can treat a verified existing invoice for the same order and amount as successful completion. Creating the invoice and any related database rows must use a transaction so a partial attempt does not appear complete. A conflicting invoice is an error to resolve, not a duplicate to ignore blindly.

This is idempotent handling: applying the same request repeatedly leaves the same intended result as applying it once. A stable identifier helps only when the destination uses it to enforce that behavior. If invoice creation also sends an email, the database constraint alone does not prevent repeated emails.

Producer idempotence has a different scope. It can prevent duplicate Kafka writes from producer-client retries, but it cannot stop an invoicing consumer from rereading offset 42. Kafka transactions can coordinate supported Kafka operations; they do not automatically include this independent invoicing database.

The input must also outlast recovery. If a backlog grows beyond the topic’s retention window, Kafka can delete records that this group still needs. A compacted topic can remove older values for a key, so retaining the latest state is not equivalent to preserving every business event. Storage and cleanup policies must match the records the application promises to process.

Watch processing failures, commit failures, repeated attempts, backlog age, and destination availability together. Rising lag during a database outage can be evidence that the consumer is correctly preserving unfinished work. A sudden drop after the consumer skips failed records would make the graph look better while leaving invoices missing.

No finite retry count turns a permanent failure into success. If the application exhausts its automatic retries, preserve the record and make the unresolved work visible. At-least-once processing has no completion-time promise, but a production application still needs one: someone must know when an order has waited too long and how to recover it.

Finally, this consumer guarantee starts with records successfully available in Kafka. It cannot recover an order event the source never published. Reliable publication, retained input, safe processing, and operational recovery all contribute to the complete business outcome.

At-least-once processing completes required work before committing the next Kafka position. Failures can repeat successful work, but recovery must not advance past unfinished records.

Skipping a commit does not rewind a live consumer. Failed records need an explicit retry or recovery path, and batch or parallel processing must preserve gaps in completion. Rebalances and uncertain commit results can also cause repeated attempts.

Use destination-side duplicate protection when repetition would cause harm. Keep input available long enough for recovery, and make unresolved failures visible so retained work eventually becomes completed work.