Database Change Data Capture

An inventory service saves a new stock quantity in PostgreSQL. A reporting database and a search index need the same update, but neither is part of that database transaction. The application could publish a Kafka record after every write, although a crash between saving the row and publishing the record could leave those copies out of date.

Change data capture (CDC) makes database changes available to other systems. In a log-based CDC pipeline, a connector reads the database's change stream and publishes records to Kafka, where consumers can apply them independently.

In this chapter, we'll follow an inventory change from the database log to Kafka, explain how existing rows join the stream, and examine the conditions needed for recovery. We'll use PostgreSQL to make the database details concrete and identify connector-specific behavior where it matters.