An order service publishes an event, and a reporting consumer adds the order amount to a daily total. Most of the time, this works as expected. Then the consumer crashes just after updating the database. When it restarts, it reads the event again and adds the amount a second time.

Kafka still has the record. The database is available. Yet the report is wrong.

Message delivery guarantees describe what can happen to a message as a system sends it, processes it, and recovers from failures. In this chapter, we’ll compare at-most-once, at-least-once, and exactly-once semantics, then follow an order event through the points where the application can lose or repeat work.

The examples use ordinary Kafka consumer groups in a KRaft-based cluster. Unless stated otherwise, records remain available during recovery, committed offsets remain valid, and consumers resume from those offsets without a manual reset.

Before choosing a guarantee, decide what counts as completion. An event reaching Kafka and its effect reaching a database are different outcomes.

Suppose the order service publishes to orders.placed with key ord-1042. The value contains:

The amount uses the currency’s minor unit: 12500 paise, or ₹125.00. Assume the record lands in partition 0 at offset 42. A consumer in the order-reporting group reads it and increases the total for that date and currency.

There are several distinct steps in this flow. The diagram separates the record’s path from the signals that report progress.

A produce acknowledgment confirms a write according to the producer’s acknowledgment settings. It does not confirm that reporting has processed the event. A fetch returns records to the consumer, but does not prove that the database changed.

An offset commit saves the group’s restart position. After completing offset 42 and all earlier records in partition 0, the consumer can commit 43, meaning that recovery should continue from there. Kafka trusts the position the application commits; it does not inspect the reporting database.

The database update is a side effect: a change outside the consumer’s local computation. Sending an email or calling a payment service would also be a side effect. These actions need their own completion rules.

For this application, the useful requirement is precise: each accepted order should contribute to the reporting total once. Saying only “the destination received the message” leaves too much unspecified.

The words at most, at least, and exactly describe how many times delivery or processing can take effect within a stated boundary. They are not three universal modes that one setting applies to an entire application.

At-most-once accepts missing work to avoid repeating it. For example, a consumer can save its progress before attempting the database update. If it crashes in between, recovery skips the order. This describes recovery for that Kafka record; it does not eliminate duplicate business events producers have already published as separate records.

At-least-once accepts repeated attempts to avoid skipping unfinished work. The reporting consumer updates the database first and then saves its progress. If it crashes between those operations, recovery repeats the record. The application must make that repetition safe if double counting is unacceptable.

Exactly-once concerns the committed result. A process may crash, restart, and execute a calculation again. The guarantee is that recovery leaves one committed effect for each input within the supported scope, rather than exposing an extra result from the failed attempt.

All three need assumptions. At-least-once processing cannot make a permanently broken handler succeed or recover an event with no surviving copies. It requires available input, eventual recovery, and a retry policy that preserves unfinished work. Likewise, none of these guarantees promises a particular completion time.

Failures are difficult partly because a caller may not know whether an operation succeeded.

Suppose the producer sends evt-7001. Kafka appends the record at offset 42, but the connection fails before the producer receives the acknowledgment. From the producer’s perspective, the outcome is uncertain: the request might have failed before the write, or the producer might have lost the response after the write.

The diagram shows the second case. For this example only, assume the producer does not use idempotence and no other records arrive between the two writes.

The retry protects against a missing write when the first attempt failed. Without duplicate protection, it can also create two records for the same event when the first attempt succeeded. Giving up avoids that retry, but leaves the application unable to promise that the event reached Kafka.

Kafka’s idempotent producer prevents duplicate log entries that the producer client’s retries can create. Here, an idempotent retry would not append a second copy at offset 43. The diagram deliberately disables this protection to expose the underlying problem; it is not a claim about current client defaults.

Producer idempotence does not compare event values or enforce uniqueness of eventId. If application code independently publishes evt-7001 twice, those sends can still become separate records. Reusing the key ord-1042 does not make Kafka reject the second write either.

This gives us two different kinds of repetition to watch for:

Producer idempotence addresses client retry duplicates. Consumer recovery can still repeat a record that exists only once in the log.

Return to the original case: evt-7001 exists only at offset 42. The reporting group has committed 42, so that record is next to process. Its daily total is currently ₹1,000.00 and should become ₹1,125.00.

The consumer must update the database and save its Kafka progress. When these are independent operations, either order leaves a failure window: a period between the two operations where a crash changes the recovery outcome.

Commit Before Processing

The consumer first commits 43, waits for confirmation, and then attempts the database update. If it crashes after the commit but before the update, its replacement resumes at 43. The total stays at ₹1,000.00.

The event is still in Kafka, but normal recovery skips it for this group. The loss is missing application work, not necessarily a missing log record. A deliberate replay could revisit it, but that is outside this recovery policy.

This order can support at-most-once processing attempts only if the application also avoids retrying uncertain processing operations. Merely placing the commit first does not prevent a separate database or HTTP retry from repeating an effect.

Process Before Committing

The consumer first commits the database update, raising the total to ₹1,125.00, and then commits Kafka offset 43. If it crashes after the update but before the offset commit, its replacement resumes at 42.

If the handler simply adds the amount again, the total becomes ₹1,250.00. Kafka has not duplicated the stored record. The consumer has repeated its effect.

These two crash points explain the trade-off:

Changing the order moves the failure window. It does not make the database update and Kafka offset commit one indivisible operation. Committing after every record can reduce how much work recovery repeats, but the window still exists.

Rebalances and Batches

A process does not need to crash for the application to repeat work. During a rebalance, partition ownership changes between group members. A replacement can resume from a committed offset that precedes work the previous consumer already completed. Losing partition ownership does not undo an external database update.

Batching adds another concern. Suppose a consumer reads offsets 42 through 49 and processes them concurrently. If 42 is unfinished while the others complete, committing 50 skips unfinished work on recovery. To preserve at-least-once processing, the saved position must not advance past that gap.

Commit the position immediately after the last record for which all earlier work has also finished, not simply the position the consumer reaches by fetching records. Automatic offset saving cannot determine whether work the application hands to another thread has finished.

Avoiding both missing and repeated effects requires more than choosing which independent operation happens first. The application needs a way to coordinate completion or recognize work that already succeeded.

Processing Between Kafka Topics

Suppose reporting instead reads orders.placed and writes a derived event to orders.reporting. Kafka transactions can include the output records and the input group’s offsets in one atomic commit: Kafka commits them together or aborts them together.

If the transaction aborts, the input remains eligible for recovery and consumers configured with isolation.level=read_committed do not receive its aborted output. If it commits, Kafka commits the output and the saved input position advances with it.

With correct transaction and recovery handling, this supports exactly-once processing effects between Kafka topics. The transformation code may still run again after a failure. Also, read_committed controls visibility of transactional records; it does not prevent a downstream consumer from rereading committed records during its own recovery.

The guarantee applies per input record. If the source published the same business event as two distinct records, a correct transactional processor can produce one result for each. Business event deduplication remains a separate requirement.

Processing Into an External Database

The reporting database does not automatically participate in a Kafka transaction. Wrapping Kafka operations in a transaction cannot roll back an independent database update or an email the application has already sent.

For the daily total, one possible design is to store a processed-event marker for evt-7001 in the same database transaction as the total update. A uniqueness constraint on the marker ensures that only the first successful handling applies the contribution. The consumer commits its Kafka offset after that database transaction succeeds.

If the consumer crashes before committing its Kafka offset, recovery presents evt-7001 again. The database recognizes the marker and leaves the total at ₹1,125.00. If the database transaction failed, the database should have committed neither the marker nor the contribution, so retry can apply both.

The marker and update must succeed together, and the destination must handle concurrent attempts safely. A separate “already processed” check followed by an unprotected update leaves another failure window. The marker must also remain available for as long as the application can replay that event.

This is idempotent processing: repeating an input leaves the same intended result as applying it once. It combines repeatable delivery with protection at the destination. It does not extend that protection to another side effect, such as sending a notification, unless that operation has its own safe retry mechanism.

An end-to-end guarantee covers the whole path from the source action to the intended destination effect. Every step has to preserve the requirement.

Suppose the order service saves ord-1042 in its own database and crashes before publishing evt-7001. A perfectly implemented reporting consumer cannot count an event it never receives. The source needs a durable way to track and retry publication of accepted orders; producer settings alone cannot close that gap.

Once the event reaches Kafka, durability and retention still matter. Durability concerns whether stored data survives failures. Delivery semantics concern how sending and processing behave during retries and recovery. Replication can protect the record while leaving the double-counting problem entirely unchanged.

Similarly, at-least-once processing depends on recovering before required records disappear. If reporting remains offline beyond the topic’s retention window, resuming from a later available offset cannot restore missing contributions. A retention policy must fit the recovery requirement.

For the reporting application, a useful design decision starts with the cost of each failure. Missing an order understates revenue; counting it twice overstates revenue. Processing before committing, combined with a database transaction that rejects repeated event IDs, addresses both consumer recovery outcomes under the assumptions described above.

A feed of replaceable device readings might accept occasional missing samples and use at-most-once handling. A Kafka-to-Kafka transformation that must coordinate output with input progress may use transactions. These choices follow from what the application needs to preserve and what its destination can support.

Delivery guarantees also do not replace ordering requirements. Two different order updates can each take effect once and still produce the wrong final state if the application applies them in the wrong order. Decide both how the application handles repetition and which operations must remain ordered.

At-most-once handling accepts missing work. At-least-once handling accepts repeated attempts. Exactly-once semantics require one committed effect per input within a clearly defined scope, even when execution repeats during recovery.

Producer acknowledgments, consumer reads, offset commits, and external effects confirm different things. Producer idempotence protects against client retry duplicates; Kafka transactions can coordinate Kafka outputs and input offsets. External destinations need their own coordination or duplicate protection.

Start with the result the application must preserve, then follow an event through publication, processing, and recovery. At each failure point, identify what has already succeeded and what a retry would repeat.