Workers, Connectors, and Tasks

A reporting pipeline is falling behind. It runs on three Kafka Connect workers, uses one sink connector, and has three tasks. The team considers adding a fourth worker, but that change may leave the pipeline with exactly the same amount of work happening in parallel.

To understand why, we need to separate the process that hosts an integration, the component that defines its work, and the components that move its records.

In this chapter, we'll look at workers, connectors, and tasks through an inventory reporting pipeline. We'll use Kafka Connect 4.3 in distributed mode with a KRaft-based Kafka cluster, focusing on how Connect creates, places, scales, and recovers work.