An inventory pipeline is running across several Kafka Connect processes. One reads database changes, another writes records to a reporting database, and a third has capacity to take on more work. If a process disappears, the others need to know which work to take over, what configuration to use, and where to resume.

Kafka Connect's architecture brings those responsibilities together. Workers execute integrations, Kafka stores shared state, and a coordination mechanism assigns work across the workers. The records themselves follow a separate path through producers, topics, and consumers.

In this chapter, we'll trace both the management path and the record path. We'll use Kafka Connect 4.3 in distributed mode with a KRaft-based Kafka cluster. The example has three workers to make placement and recovery visible; it is not a production sizing recommendation.

A Connect worker is a process that runs connector code and the runtime services it needs. Several workers can form a Connect cluster, sharing responsibility for a set of integrations.

That cluster is separate from the Kafka cluster. Connect workers read external systems, transform and convert records, and write destinations. Kafka brokers store topic records and serve Kafka client requests. KRaft controllers manage Kafka's cluster metadata, including broker and partition leadership information.

For our example, inventory-source reads inventory changes and publishes to inventory.stock. The inventory-reporting sink writes those records to a reporting database.

The diagram shows one possible placement of their data-transfer tasks. Each worker also runs management components, which are omitted here. Dashed lines represent access to shared Connect state in Kafka.

Worker 1 sends records to Kafka, and Worker 2 reads them from Kafka. They do not transfer inventory records directly to each other. Moving the sink task to Worker 3 changes where the destination writes run, while the topic remains the same.

Connect workers are Kafka clients. Adding a worker adds execution capacity for integrations; it does not add broker storage or replicas. Likewise, adding a broker does not install or start connector code.

A worker combines framework code with installed plugins. The framework manages execution and Kafka communication. A plugin supplies specialized behavior, such as reading a database, encoding a value, or writing a search document.

Within that runtime, a connector instance manages a named integration and produces configurations for its tasks, the units that perform the data transfer. The connector instance and its tasks can run on different workers. A task does not stream its records through the worker that hosts the connector instance.

The worker supplies the Kafka producer or consumer path around each task, along with conversion, optional transformations, progress tracking, and status reporting. This lets a plugin concentrate on the external system's protocol and data model.

Plugins execute inside the worker process. Separate plugin class loaders help keep library dependencies apart, but they do not provide separate operating-system processes or independent memory limits. A memory problem in one integration can therefore affect other work the same worker hosts.

The required plugins and dependencies must be available on every worker that may receive the work. Kafka stores connector configurations, not the plugin binaries. A replacement worker can recover a configuration from Kafka and still fail to start it because a required plugin is missing.

This is why worker preparation matters before assignment: installing a plugin on the worker that currently runs a task is insufficient if the task can later move elsewhere.

Connect has a control plane, the mechanisms that accept configuration, decide where work runs, and report its state. Its data plane is the execution path that moves records. Both run within the worker service, but they serve different purposes.

Each worker exposes a REST API. An operator can submit a connector configuration to a worker without choosing the worker that will execute it. Requests that require another worker, such as a configuration write that the current Connect leader handles, can travel between workers internally.

The distributed workers join a group identified by the worker setting group.id. A Kafka broker acts as that group's coordinator, managing membership. One worker becomes the Connect group leader and computes assignments of connectors and tasks using Connect's assignment logic. The broker coordinator distributes the group assignment through the group protocol.

The Connect leader is an ordinary worker with an additional management role. It can also run tasks. It is distinct from both the broker coordinating the group and Kafka's active KRaft controller.

From Configuration to Execution

Suppose the team creates inventory-reporting. The sequence below summarizes how the configuration becomes running work. It omits the individual group-protocol messages and any intermediate assignment rounds.

Connect saves the connector configuration, the connector generates task configurations, and Connect assigns the resulting work. Workers load the necessary plugins and start their assignments. The whole process can take multiple steps; accepting the configuration does not mean the reporting database already contains inventory data.

Workers also consume status updates asynchronously. A status response can briefly trail a state change, and the connector's state can differ from a task's state. For example, the connector may be running while a sink task has failed to write to its destination.

Two Kinds of Group Membership

The Connect worker group answers: which worker runs each connector and task?

The reporting sink's consumer group answers: which sink consumer reads each Kafka topic partition?

These are separate groups with separate assignments. Moving a sink task between workers changes its execution location and can also change membership in the sink consumer group. The topic-partition assignment then follows the consumer group's rules.

Workers intended to form one Connect cluster need the same worker group.id and consistent shared-state topic settings. Independent Connect clusters should use distinct group identities and separate internal topics. Pointing two unrelated groups at the same configuration topic does not safely isolate their integrations.

Distributed Connect stores the information needed to manage and resume integrations in Kafka topics. A worker keeps local runtime state and cached information, but its memory is not the only record of what the cluster should run.

Three worker settings identify Connect's main internal topics. Our example uses the worker's shared source-offset topic and keeps both internal topics and business topics in the same Kafka cluster. The names in this table are examples we chose for our deployment, not built-in topic names.

These topics use log compaction, which allows Kafka to retain the latest value for a key while eventually removing superseded values. The configuration topic requires one partition so configuration changes have a single ordering. The source-offset and status topics can have multiple partitions.

Their replication and availability matter because workers depend on them for management and recovery. They are operational state, not disposable logs. Deleting a source-offset topic, for example, can remove the checkpoint a source needs to distinguish previously captured input from new input.

Source Offsets and Sink Offsets

The Connect source-offset topic does not normally hold the reporting sink's consumer offsets. Those belong to the sink consumer group and Kafka stores them in __consumer_offsets.

For an inventory update at partition 0, offset 120, there are several distinct pieces of state:

Keeping these separate explains what recovery can reconstruct. A saved connector configuration tells a worker how to start a pipeline. A saved source checkpoint or consumer offset tells it where to resume. Neither substitutes for the actual input records or the destination's data.

Once Connect assigns the work, records travel through the worker's data plane. They do not pass through the REST API, the Connect group leader, or the configuration topic as an intermediate forwarding step.

Assume inventory-source captures a stock update with key sku-1042 and logical value {"availableUnits":27,"version":86}. The application maintains the version. The source preserves the key, and Kafka stores the record in inventory.stock, partition 0, at offset 120.

The diagram expands the source and sink paths. Single message transforms (SMTs) are optional plugins that modify individual Connect records. Converters translate keys and values between Connect's data representation and serialized bytes.

On the source side, transformations run before conversion to bytes. On the sink side, conversion from bytes happens before transformations. This gives transformations structured Connect data to work with rather than an arbitrary serialized payload.

The producer and consumer communicate with the brokers serving the topic partitions. Even if both tasks happen to run on the same worker, this source-to-sink pipeline still passes through Kafka.

The reporting task receives the decoded stock update and writes the destination row. Under our assumed at-least-once sink behavior, once the sink has successfully handled offset 120 and all preceding records, Connect can commit 121 as the sink group's next position for that partition.

A sink-side transform changes what that sink task receives; it does not rewrite the stored record at offset 120. Another consumer of inventory.stock still reads the original topic data. To publish a transformed stream for other applications, the system needs an explicit write to a Kafka topic.

Some converters use an external Schema Registry service to obtain schemas. That adds another dependency on the conversion path, but Schema Registry is not a required component of every Connect deployment. The dependency follows from the chosen format and converter.

Suppose Worker 2 crashes while running the reporting sink task. The worker loses its in-memory buffers and active connections. The inventory records, shared configurations, and committed progress remain in Kafka, assuming the Kafka cluster retains and can serve them.

The Connect worker group detects the missing member. After the applicable failure-detection and rebalance delays, Connect can reassign work. Suppose Worker 3 receives the reporting task and has the required plugin, credentials, network access, and capacity.

Worker 3 starts the task from the shared configuration. Its sink consumer joins the reporting consumer group, receives a topic-partition assignment, and resumes from committed offsets. It reconnects to the reporting database and continues writing.

This is recovery by restarting work from durable state. Connect does not copy Worker 2's live memory to Worker 3, and the extra worker is not a replica of the sink's in-flight writes.

If Worker 2 wrote offset 120 to the database but crashed before committing 121, the replacement can apply offset 120 again. The sink's write behavior must make that repetition safe. A Connect rebalance does not make the database write and Kafka offset commit atomic.

The sequence below shows this recovery path from the crash to the resumed writes.

Modern Connect supports incremental cooperative rebalancing, which can move affected work while leaving unrelated assignments running. Recovery time still depends on group settings, startup time, and external connections. Do not expect instantaneous failover.

If the departing worker was also the Connect group leader, another worker can assume that management role through group coordination. This does not change who leads Kafka's topic partitions; broker leadership is a separate concern.

Task Failure Within a Live Worker

A task can fail while its worker remains healthy, such as when a destination rejects an incompatible record. That is different from losing a group member. In the base Connect runtime, Connect does not automatically restart a task in FAILED merely because another worker is available. An operator or external automation must address the failure and request a restart as needed.

This distinction helps interpret status. A responsive REST API proves that the management endpoint is reachable. It does not prove that tasks are processing records, that converters can decode them, or that the reporting database is current.

When the report stops updating, trace the pipeline: check task status, whether source records reach the topic, whether the sink's offsets advance, and whether destination writes succeed. Each observation narrows the problem to a specific part of the pipeline.

Kafka Connect workers execute integrations while Kafka stores business records and shared management state. The worker group assigns connectors and tasks; sink consumer groups separately assign topic partitions and track consumption progress.

Configuration and status flow through the management path. Records flow through tasks, transformations, converters, and Kafka clients. Durable state lets work resume on another worker, while connector behavior and external-system capabilities determine whether retries produce the correct destination result.