A dashboard shows the current temperature in an office. Sensors publish a new reading every few seconds, and a consumer sends those readings to the display service. If one update fails, the next reading can replace it. The dashboard does not need to retry every old sample to remain useful.

At-most-once processing makes that trade-off explicit: a record gets at most one processing attempt, and the application may never process some records. For a Kafka consumer, the central decision is to save progress before beginning the work.

In this chapter, we’ll follow that sequence through crashes, failed commits, and rebalances. We’ll also build a small Java processing loop and examine when accepting missing work is reasonable.

An at-most-once processing policy allows zero or one invocation of the application’s handler for each Kafka record within a consumer group’s normal recovery path. It does not promise that the invocation succeeds or that every record receives an attempt.

For our dashboard, the handler sends a temperature update to a display service. A handler is the application code that performs the work for a record. Here, it makes one request and does not retry if the result is uncertain.

Suppose office.temperature contains a record with key sensor-17 in partition 0 at offset 42. Its value is:

The consumer group is office-live-display. Before reading this record, its committed position for partition 0 is 42. We’ll assume the group resumes from its saved offsets after a failure, those offsets remain valid, and no application or administrator moves them backward.

The scope matters. If a producer publishes reading-9001 again as a separate Kafka record, both records can receive a processing attempt. At-most-once consumption does not deduplicate business events. Another consumer group can also process offset 42 independently.

Even within one group, receiving bytes and invoking the handler are different events. A consumer can fetch offset 42, crash before saving progress or doing any work, and fetch it again after restarting. That repeated fetch does not violate our processing contract because the handler has not yet run.

For one record at offset 42, the sequence is:

The commit is deliberately ahead of the work. Kafka will resume the group after this record even if the application never finishes processing it.

The diagram shows the success path and the point at which a crash can leave the reading undisplayed.

Once the commit succeeds, normal recovery cannot use offset 42 to finish the missing update. Kafka may still retain the record, but the group has already advanced beyond it.

Do not always describe an offset commit as “acknowledging completed work.” It saves a restart position. The application chooses whether that position represents completed work or work it has decided not to revisit.

A producer acknowledgment has a different purpose: it tells the sender about the write to Kafka. It does not determine when this consumer saves progress or whether the temperature reaches the display.

Waiting for commit confirmation is essential. Sending a commit request is not enough.

Suppose the consumer requests offset 43, then calls the display service before learning the commit’s outcome. The display might update successfully while the commit fails. Recovery would resume at 42 and invoke the handler again. The application has broken its intended at-most-once policy.

For this design, the rule is simple: do not start processing unless the required offset commit has succeeded.

Here are the possible outcomes for our single-record example:

The first two cases can look the same to the consumer: a timeout with an unknown result. The safe choice for this processing policy is to withhold the work. If Kafka saved the offset, that choice loses the update. If Kafka did not save it, a replacement consumer can read it and attempt the commit again before processing.

Retrying a commit before any processing has started is different from retrying a side effect. A commit retry does not itself update the display. However, recovery code must still respect current partition ownership and must not submit stale offsets that move progress backward. A simple implementation can stop on commit failure and let a fresh consumer recover instead.

The display request has its own uncertainty. A timeout might mean the service never received it, or that the service applied it but the consumer lost its response. Retrying that request could repeat the effect. To retain an at-most-one-request policy, the application must also account for retries inside HTTP clients, SDKs, and other wrappers around the handler.

Kafka cannot enforce how often an external service executes a request. The guarantee we control here is one handler invocation per record, with no application retry of an uncertain effect.

The following class illustrates the ordering with Java 21 and org.apache.kafka:kafka-clients:4.3.1. It is a helper for an existing Java application, not a standalone command-line program. The caller supplies connection properties and the display handler.

Use a reachable KRaft-based Kafka cluster where you have already created office.temperature. The supplied properties must include bootstrap.servers and group.id=office-live-display, plus any authentication and encryption settings the cluster requires. For a local learning broker, bootstrap.servers=localhost:9092 may be appropriate for that broker’s advertised listener.

This example sets auto.offset.reset=none: the group must already have a valid starting offset for every assigned partition. If it does not, the application fails so an operator can choose a starting position deliberately. This keeps an implicit reset from changing the recovery behavior we are examining.

The class disables automatic commits and processes each batch only after Kafka commits its offsets. The supplied handler must finish its single attempt synchronously, use bounded request timeouts, and avoid internal retries.

records.nextOffsets() supplies the positions after the returned records, separately for each partition, including available leader-epoch metadata. For a batch containing only partition 0, offsets 42 through 46, it provides the next position 47. Using the batch’s returned metadata also avoids assuming that all Kafka offsets are consecutive.

commitSync waits for commit completion or throws. Its failure is intentionally outside the handler’s catch block: the method exits and closes the consumer without attempting any record in that batch. Restarting must use a fresh consumer and its saved group positions, not resubmit the old in-memory batch to the handler.

A handler exception has a different policy. The saved position is already beyond this record, so the example reports the failed or uncertain attempt and continues. It does not seek backward or queue the record for another attempt. The log contains the Kafka position rather than the event payload, which is enough to identify the affected record without exposing its contents.

The one-second poll duration controls how long the call may wait for records. It is not a processing deadline. The application must keep the total commit and handling time within its consumer liveness limits. This helper omits service startup and coordinated shutdown; an abrupt stop can abandon the remainder of a committed batch, which is part of this policy’s loss risk.

Committing an entire batch before processing reduces the number of commit requests, but it puts every record in that batch beyond the recovery position at once.

Suppose the consumer receives offsets 42 through 46 from partition 0, with one temperature reading at each offset. It commits 47, successfully handles 42 and 43, and then crashes.

The diagram shows what survives in group progress and what remains unfinished:

This group skipped three readings even though Kafka may still hold all five. A crash immediately after the commit could abandon all five.

A smaller max.poll.records can reduce the number of returned records a crash can skip after one batch commit in this sequential loop. It does not eliminate the gap between the commit and the next handler invocation. The setting limits records poll returns; it is not a direct limit on all data the client may have fetched internally.

Committing a separate next position before each record narrows the exposure further, at the cost of more commit requests and waiting. Take care with the no-argument commitSync(): it commits the positions from the last poll, so putting it inside a per-record loop does not make it a per-record commit.

There is no universally correct batch size. For the dashboard, losing several replaceable readings may be acceptable. If one missing record is unacceptable, shrinking the batch is not enough; the processing policy must change.

A rebalance transfers partitions between members of a consumer group. With progress committed before processing, a replacement starts after the committed batch. It does not know which handlers the old consumer actually finished.

If ownership changes before the commit can succeed, the old consumer must not process that batch. A group-related commit failure is not a reason to ignore the error and continue with the records it already fetched.

If ownership changes after the commit succeeds, the new owner skips the committed records. The previous owner may still have an external request in progress. Kafka cannot cancel that request, and the new owner may begin handling newer readings before it finishes. Even without repeated record processing, the display can receive an older reading after a newer one. Comparing observation timestamps at the destination is one way to prevent the display from moving backward in time.

Several settings and shortcuts are easy to mistake for an at-most-once policy:

Automatic commits can produce different failure outcomes depending on how the application polls and handles records. Do not describe automatic commits as automatically providing at-most-once processing.

The defining property is the application’s execution order: confirmed progress first, then one attempt at the corresponding work.

At-most-once processing is most useful when missing a record has a known, acceptable consequence. Our office dashboard shows replaceable state. If it misses 21.3°C, a later reading of 21.4°C can still make the display useful without reconstructing every intermediate sample.

The same temperature topic could serve a safety application that must detect every threshold crossing. Missing the only reading above a dangerous threshold would be unacceptable. The data source is the same, but the application’s requirement is different.

This distinction is also important for totals and deltas. A later complete reading can replace a record saying “current occupancy is 18”. A record saying “three people entered” contributes to accumulated state; skipping it can leave the count wrong indefinitely. Financial entries, audit events, and inventory adjustments usually need that stronger accounting of individual events.

Accepting loss does not remove the need to detect it. In this design, committed offsets advance before useful work. A dashboard based only on committed-offset lag can look healthy while handlers are failing.

The diagram separates saved Kafka progress from the application result that users care about:

For the dashboard, useful signals include successful update counts, failed or uncertain attempts, and the age of the latest reading the display actually shows for each sensor. A timestamp that stops advancing reveals a stale display even when the consumer has committed every fetched offset.

Application logs alone cannot account for every lost record: a process may crash after committing and before logging an attempt. Exact loss measurement requires independent evidence, such as source sequence numbers and observations at the destination. That additional accounting may cost more than this best-effort display needs, but understand this limitation.

Finally, at-most-once does not imply that processing is automatically fast. Waiting for a commit adds work before the handler can begin. The benefit is a deliberate recovery policy that avoids reattempting old work; whether it improves performance depends on batching, request costs, and the workload.

At-most-once processing commits a record’s next position before invoking its handler and waits for that commit to succeed. Recovery can then skip unfinished work, so some records may receive no attempt at all.

Failed or uncertain commits must prevent processing from starting. Handler retries, duplicate publications, offset resets, and external service behavior each affect the scope of the guarantee. Batch commits increase how much work a crash can abandon.

Use this policy when missing updates is acceptable, such as a display of replaceable readings. Monitor successful effects and freshness as well as saved Kafka progress, because a committed offset does not prove the work happened.