Designing Idempotent Consumers

A reporting consumer updates a daily sales total, then crashes before committing its Kafka offset. When it restarts, Kafka returns the same order again. Retrying is necessary for reliable recovery, but adding the amount twice would make the report wrong.

An idempotent consumer makes that repeated attempt safe. It either performs an operation that naturally leaves the same result or recognizes that the required effect already happened.

In this chapter, we’ll design a database-backed consumer that survives crashes and concurrent retries. We’ll also examine event identity, stale updates, external services, and the lifetime of duplicate protection. The Kafka examples use ordinary consumer groups in a KRaft-based cluster; the SQL uses PostgreSQL 17 with its default Read Committed isolation level.