Log Compaction

An inventory service publishes the available quantity of each product whenever it changes. A new application needs the current quantity of every product, including products that have not changed for months. Deleting all old records would remove some of those unchanged values, while keeping every update forever would accumulate unnecessary history.

Log compaction lets Kafka remove superseded values while preserving the information needed to rebuild current state. We'll follow a small inventory log through compaction, examine deletion markers, and see what producers and consumers must do to make state recovery reliable.

Our examples use a KRaft-based cluster with local storage and tiered storage disabled. The topic inventory.wh1 describes inventory in warehouse 1 and uses cleanup.policy=compact. Records are non-transactional and committed. Producers consistently route each product’s updates to one partition. Unless explicitly discussed, no delete cleanup policy or administrative deletion applies.