Building Reliable Data Pipelines

An inventory reporting pipeline works well during testing. Database changes appear in Kafka, a sink connector updates the reporting database, and the dashboard shows the right quantities. Then the reporting database becomes unavailable for an hour. When it returns, the connector starts writing again, but some records arrive twice and the backlog keeps growing.

A reliable pipeline needs to handle this entire sequence. Moving records during normal operation is only part of the job. The pipeline must preserve their meaning through retries, expose missing or delayed data, and provide a way to recover when something goes wrong.

In this chapter, we'll use an inventory reporting pipeline to connect these concerns into a practical design.