Imagine an online store that keeps its product catalog in PostgreSQL. Customers search for products through a search index, and an analytics team needs a copy of product data for reporting. When a product's price or description changes, those other systems need to catch up.

You could write a program that reads database changes and publishes them to Kafka, then write consumers that update each destination. But moving the first record is only part of the job. Those programs also need to track progress, recover from failures, handle incompatible data, and keep running as the catalog grows.

Kafka Connect is Apache Kafka's framework for moving data between Kafka and other systems. In this chapter, we'll look at the integration work it handles, follow a product update through a pipeline, and explain which decisions still belong to your team.

A data pipeline moves data through a sequence of systems. In our store, one pipeline starts at the product database, passes through Kafka, and ends at the search index.

A custom implementation might look straightforward: read changes, publish records, consume them, and write documents. Once it runs continuously, several details become essential.

The database reader needs a reliable way to find new changes. It must remember how far it has read so that a restart does not require copying the entire catalog again. The search writer needs to save its Kafka offsets at the right time and handle a destination that is temporarily unavailable. Both need compatible data formats and a way for operators to see when work has stopped.

Now add a second database or another destination. Much of the operational code repeats, even though the system-specific reading and writing logic differs.

Kafka Connect separates these concerns. The framework supplies a common runtime for integrations, while installable connector plugins provide the logic for particular systems. A database connector knows how to read its database; a search connector knows how to write to its search service.

When a suitable plugin exists, you can configure an integration instead of building and maintaining its entire runtime yourself.

Connect supports two directions of data movement. A source connector brings data into Kafka. A sink connector takes data from Kafka and writes it elsewhere. The names describe the direction relative to Kafka.

For the store, a database source publishes product changes, while separate sinks feed search and analytics. The diagram shows the logical integrations; the connectors run inside Connect processes.

Kafka stores the records between the source and the destinations. With separate consumer groups, the two sinks maintain independent progress. A search outage can delay search updates while analytics continues processing, provided Kafka and the rest of the pipeline remain available.

Connect uses Kafka producers and consumers underneath. The records it writes are ordinary Kafka records, so an application can consume them without using Connect. Likewise, a sink can read records that an application's producer publishes. A pipeline does not need a connector at both ends.

Connect runs separately from Kafka brokers. Brokers store and serve records; Connect processes run the integration code and communicate with external systems. Starting a Kafka cluster does not automatically start your database or search integrations.

A plugin is reusable code. A connector instance is a named integration that uses that code and a particular configuration. For example, two instances could use the same database plugin to read from different databases.

A worker is a Connect process. The connector instance defines the work, and tasks perform the actual reading or writing on workers. This division lets the framework manage execution without needing to understand each external system's protocol.

To put our catalog integration into service, the team would:

Connect provides a REST API for managing connector instances and inspecting their status. The settings for reading database changes or writing search documents come from the selected plugins, so there is no universal configuration that works for every database and destination.

Connect can run in standalone mode in a single worker, or in distributed mode with workers sharing the work. Distributed mode can reassign work after a worker fails. That recovery still requires available workers with the right plugins, network access, and credentials.

Adding workers also does not guarantee more throughput. The connector must be able to divide its work, and the external system must be able to keep up.

Suppose a catalog administrator changes the price of product prod-1042 to $79.99. For this example, assume the database source supports change data capture (CDC), which detects database changes and makes them available to other systems. Here, it reads committed changes from the database's change log.

Not every database source works this way. Some query tables periodically. The capture method determines which changes the connector can observe and how quickly it can publish them.

Assume we configure our pipeline to publish each product's current fields to catalog.products, using its product ID as the record key. The following JSON illustrates the value after any configured record shaping. It is an example data contract, not the default output of every CDC connector.

Kafka stores the key, prod-1042, separately from the value. The application maintains version as an increasing number for each product; Connect does not invent that version. A converter handles conversion between Connect's data representation and the bytes Kafka stores. Its configuration must match the record format readers expect.

The diagram follows this update through the search path. Assume Kafka assigns it offset 42 in partition 0.

The source tracks a position in the database's change stream. The sink tracks positions in Kafka partitions. These are different checkpoints: the database position identifies what the source has captured, while Kafka offset 42 identifies where Kafka stored this record.

Assume the search sink maps the Kafka key to the document ID and supports replacing an existing document. It writes the fields to document prod-1042. Once the sink has successfully handled offset 42 and all earlier records in that partition, it can commit 43 as the next Kafka position to read.

The database commit, Kafka publication, and search update occur at different times. A successful database update therefore does not mean customers can immediately see the new price in search. The search view catches up as the pipeline processes the change.

Saving progress lets an integration resume after an interruption, but the destination write and progress checkpoint are not automatically one atomic operation.

Consider our search sink with at-least-once processing. Assume its committed position for partition 0 is 42, so that is the next record it must read. It successfully updates the search document, then crashes before committing 43.

The sequence below shows the resulting recovery path. Assume the record remains in Kafka and the sink resumes from its committed position.

Writing the same fields to the same document ID can make this immediate retry harmless. Appending a new document with a generated ID on every attempt would instead create duplicates.

The stable ID alone does not make every replay safe. If version 19 has already reached the index and a consumer replays an older update, blindly replacing the document with version 18 could restore a stale price. Preventing that requires suitable ordering or version checks that the connector and destination support.

Connect supports exactly-once operation in some configurations, including source-side transactional publication. That does not create a universal transaction covering the database, Kafka, and every destination. Delivery guarantees depend on the connector, its configuration, and the external system's capabilities.

Recovery also has limits. A source cannot resume from a database log position that the database has already discarded. A sink cannot reread a Kafka record that retention has removed. Invalid credentials or incompatible records need correction; moving a task to another worker does not resolve them.

Connect is a good fit when the main job is moving data between systems and an existing connector supports the behavior you need. Examples include keeping a search index current, ingesting database changes, and delivering Kafka records to an analytics store.

The existence of a plugin is only the starting point. For our catalog, the team must check whether it captures every required change, how it represents deletes, and whether the search sink understands its output. It must also check supported versions, maintenance, licensing, and recovery behavior. Connector plugins come from different projects and vendors; Apache Kafka does not bundle all of them.

Connect supports lightweight single message transforms (SMTs), which modify individual records, such as renaming a field or removing an unnecessary field. Those transformations are useful for adapting data at an integration boundary.

Work that combines multiple events needs a different approach. Computing a five-minute sales total or joining orders with payments belongs in a stream processing application, using tools such as Kafka Streams. A consumer that decides whether to approve a refund also needs application logic beyond copying records to a destination.

Custom producers and consumers remain appropriate when the integration is closely tied to business behavior or no suitable connector exists. Building a custom connector is another option when you want system-specific code to run within Connect's framework.

Whichever approach you choose, the team still owns the data contract and the pipeline's operation. For the catalog, that means monitoring how far search trails the database, detecting failed tasks, managing access, and checking that updates and deletes produce the expected documents. A running process alone does not prove that the search results are current or correct.

Kafka Connect runs reusable integrations that move data into and out of Kafka. Source connectors bring data in, sink connectors write it to destinations, and workers execute the tasks that perform the transfers.

The framework reduces the runtime code a team must maintain, while each plugin supplies system-specific behavior. Reliable pipelines still depend on compatible data contracts, correct checkpointing, safe retries, and the capabilities of the systems at both ends.