Consumer Lag and Backpressure Tuning

An order-tracking database slows down during a traffic spike. Kafka continues accepting events, and the tracking consumers continue fetching them. The consumers appear busy, but their work queues grow until they run out of memory and restart.

The problem is not simply that records are waiting. Kafka lets consumers fall behind and catch up. The problem is allowing more work into the application than it can safely hold and finish.

In this chapter, we’ll distinguish different kinds of lag, diagnose why unfinished work is growing, and build a backpressure policy that limits intake without losing progress. We’ll also estimate recovery time and decide when a backlog threatens freshness or access to retained records.