Duplicate Message Problem

The order service accepts an order once, but the reporting system counts it twice. The producer logs show a retry. The consumer logs show a restart. Both are normal recovery actions, yet either could explain the incorrect total.

The duplicate message problem begins with this ambiguity: the same business event can appear in multiple Kafka records, and a consumer can process one Kafka record multiple times. Those cases look similar at the destination but need different explanations and protections.

In this chapter, we’ll trace where duplicates come from, distinguish record identity from event identity, and examine what makes repeated processing safe or harmful. The examples use ordinary consumer groups in a KRaft-based Kafka cluster, with Kafka retaining records and valid offsets throughout recovery.