A delivery company publishes shipment-status records to Kafka. Its customer notification service only needs to know when a shipment goes out for delivery, so a small application reads the status stream and forwards matching records to a separate topic.

You could write this application with a consumer loop and a producer. As it grows, you'd also need to manage processing threads, coordinate instances, and connect saved progress to completed output. Kafka Streams handles much of that runtime work while letting you describe the processing logic directly.

This chapter explains the main pieces of a Kafka Streams application: its processing topology, record formats, application identity, and execution model. We'll use a small Java application to connect those pieces and follow what happens when it starts, scales, and recovers.

Kafka Streams is a Java client library for processing data that Kafka stores. You add it as a dependency, write your processing logic, and run the resulting application in a Java process.

The application can run on your laptop, a virtual machine, or in a container. Kafka Streams uses Kafka clients to read input, write output, and coordinate processing. Your deployment platform remains responsible for starting processes, allocating resources, and restarting failed deployments.

The diagram shows the boundary between the Kafka cluster and the application. Both topics live in Kafka; the filtering code runs in the application's process.

Kafka Streams manages the clients this pipeline uses. You don't write a polling loop or call a producer for every forwarded record. The brokers continue storing and serving records; they don't execute your Java filter.

This deployment model also means that a working Kafka cluster is only part of a working pipeline. If every instance of the routing application stops, new status records can accumulate in Kafka, but no new routing results appear until an instance resumes processing.

A topology describes the processing steps and the connections between them. It is the plan that Kafka Streams executes for incoming records.

Our shipment router has three steps: read shipments.status, select records with the value OUT_FOR_DELIVERY, and write them to shipments.out-for-delivery.

The diagram shows the logical computation. These nodes are processing steps, not separate servers or services.

A source processor brings records from Kafka into the topology. An intermediate processor applies an operation, such as this filter. A sink processor writes results to Kafka. Here, “sink” means a topology output node; it is not a Kafka Connect sink connector.

A record that fails the filter produces no output. A matching record continues to the sink. Other topologies can branch into several paths or combine data, but the same idea applies: records move through connected processing steps.

Kafka Streams provides two main ways to express this logic. The Streams DSL, a set of Java methods for describing stream operations, supplies operations such as filtering, mapping, and aggregation. The Processor API gives you more direct control over record processing and state access. The DSL is a useful starting point for a pipeline built from standard operations.

In the DSL, StreamsBuilder collects the operations you declare, and build() produces a Topology. Defining the topology does not read records. A separate KafkaStreams object starts and manages its execution.

Kafka stores record keys and values as bytes. Your Java application needs types it can work with, so it must decode input bytes and encode output values.

A Serde combines a serializer and a deserializer. Kafka Streams uses Serdes at boundaries where data enters or leaves Kafka, and where stateful operations store data. The key and value can use different Serdes.

Our example uses a deliberately small contract. Each input record has a shipment ID as a UTF-8 string key and a status code as a UTF-8 string value. The upstream system validates the status codes. Null values are outside this input contract and produce no output in our filter.

For example, the input might contain:

This contract keeps serialization visible without introducing a JSON parser or schema tooling. A richer shipment event could include an event ID, occurrence time, and carrier details, with a Serde appropriate for its format. The processing library does not infer those fields or their meaning.

Serdes.String() handles the strings in this example. Consumed.with(...) specifies the input key and value Serdes, while Produced.with(...) specifies the output Serdes. Declaring them at the topic boundaries makes the expected types explicit.

A Serde must match the producer's encoding. Choosing a string Serde for a JSON value gives the application JSON text, not a parsed Java shipment object. Correct decoding and validation are separate concerns.

The following example uses Java 21 and Kafka Streams 4.3.1. It assumes a local KRaft-based Kafka broker reachable at localhost:9092, with no authentication or encryption, and Kafka command-line tools available from the Kafka distribution directory. These connection settings are for local learning.

In a Maven Java project configured for Java 21, add this dependency inside the project's dependencies element:

Kafka Streams brings in its required client dependencies. If the project does not already have an SLF4J logging provider, add this one for the local example so that library startup and failure messages are visible:

Use one logging provider in the application. An existing service can keep its own compatible logging setup.

Create the input and output topics explicitly. Both use four partitions for the assignment example below; replication factor 1 assumes a single local broker and provides no replica redundancy.

Save this class as src/main/java/ShipmentRouter.java. It defines the topology in a separate method so you can also inspect or test the processing plan without starting the live application.

The object that builder.stream(...) returns is a KStream<String, String>, a stream of records with string keys and values. The filter keeps the key and value unchanged for matching records. The to(...) call declares the destination topic.

topology.describe() prints the processing graph, which is useful for checking that the application reads and writes the intended topics. streams.start() begins background processing; it does not wait for all available input to finish. Run ShipmentRouter.main() through your IDE with the project's Maven dependencies on its classpath.

The exception handler reports an uncaught stream-thread failure and requests shutdown of this client. It does not repair the input, restart the Java process, or stop every other application instance. The shutdown hook gives an ordinary process termination a chance to close Kafka Streams, using a ten-second timeout for this local example. Abrupt process or machine failures can bypass the hook.

To send the sample statuses, run this producer in another terminal:

The colon separates the key from the value for this command-line producer. It is not part of either field. Read the output and display the keys:

On fresh topics, with one submission and no processing failure, the output contains shipment-1042:OUT_FOR_DELIVERY. The other two records fail the filter. Repeating the producer command submits new input records, so the router produces another matching output.

The application.id value, shipment-router, identifies the logical application. Multiple instances using that ID cooperate to process its input. A second instance shares the work; it does not independently read every input record.

Kafka Streams also uses this identity for processing progress and to give application resources a shared name prefix, such as internal topics and local state when the topology needs them. Treat it as a stable part of the application, rather than a disposable process name.

Suppose the router has completed records through offset 42 in one input partition and committed the next position, 43. An ordinary restart with the same application ID can resume from 43, provided that position remains valid. Each input partition has its own position.

A different application ID has independent progress. Starting shipment-router-preview does not inherit shipment-router's committed offsets. Its starting position follows the configured reset policy when no valid committed position exists. Kafka Streams 4.3.1 defaults that policy to earliest, so a fresh identity can read retained history.

This is useful for an independent evaluation, but it can also produce unexpected output if both identities write to the same destination. A separate application ID does not automatically create a separate output topic. For isolated processing, choose separate destinations too.

Instances sharing an ID should run compatible processing logic and configuration. Two unrelated topologies must not share an application ID just because they happen to read the same topic. Likewise, changing an ID during a deployment is a deliberate change to processing identity, not a routine way to label a software release.

The topology describes the work. Tasks divide that work according to input partitions, stream threads execute the tasks, and application instances host the threads. In this example, each instance is one running ShipmentRouter process.

Our topology reads one four-partition topic and has no repartitioning step. It therefore has four active stream tasks, one for each input partition. Every task applies the same filter to its own records.

The diagram shows one possible assignment across two instances, each configured with one stream thread. Task labels here are descriptive labels, not Kafka Streams' generated task IDs.

Each thread handles two tasks. Those tasks share the thread's processing time; they do not each have a dedicated thread. If only one instance runs, its single thread can handle all four tasks.

Adding a second instance or increasing num.stream.threads can provide more processing concurrency when resources and available tasks permit it. For this topology, more than four stream threads across the application cannot give you more than four active tasks. Extra threads have no additional active task to execute.

That limit is specific to this simple topology. More complex topologies can contain several connected processing groups, with task counts that depend on their input partitions. Avoid applying the single-topic example as a universal formula for every application.

When instances join or leave, Kafka Streams redistributes task ownership through Kafka coordination. This reassignment is a rebalance. If Instance A fails, a remaining instance can take over its tasks, provided it can reach Kafka and has capacity. More evenly distributed task counts do not necessarily mean equal load: one busy shipment partition can require much more work than another.

Kafka preserves order within each partition. Processing several partitions concurrently does not create a total order across them, and the output topic has its own partitions and offsets. An input offset is never a promise of the same offset in the output.

Our filter is stateless: it decides whether to forward a record using only that record. If the router instead maintained a count per carrier, it would need to retain earlier results. Kafka Streams provides state stores for that processing state.

State belongs to the tasks that use it. When a task moves to another instance, the destination may need to restore its state before continuing. Kafka-backed recovery normally uses changelog records or a suitable source topic, depending on the store and topology. It relies on the required data remaining available and an appropriate recovery-logging configuration.

The routing example explicitly uses at-least-once processing. If it publishes a matching record and fails before committing input progress, recovery can process that record again and publish another copy. A graceful shutdown reduces unfinished work but does not eliminate every failure window.

Kafka Streams also supports exactly-once processing for coordinated Kafka outputs, input progress, and state updates. That guarantee does not automatically cover sending a text message or updating an unrelated database. A notification service consuming our output still needs a policy for repeated status records and notification retries.

The business contract matters here too. Our small input contains a shipment ID and status, but no unique event ID. A notification service might allow only one out-for-delivery message per shipment, or it might need to distinguish separate delivery attempts. That decision requires the appropriate identity fields and durable tracking in the notification workflow.

Kafka Streams manages processing mechanics within its supported boundaries. The application still defines what a repeated business event means and which external actions are safe to repeat.

Starting the Java process is the beginning of operation. The application still needs to connect, receive task assignments, and restore state when necessary. A returned start() call does not prove that records are already flowing.

Observe the Kafka Streams runtime state, processing errors, and input lag, then verify that expected output reaches its destination. For the shipment router, a process can be alive while a broker connection problem prevents it from delivering any selected statuses.

Avoid slow external calls inside a processing callback without considering their effect. If the filter called a carrier API for every record and that API stalled, other tasks on the same stream thread would wait too. Publishing selected records to a separate topic lets an independent notification application handle its own external calls and retries.

Production deployment also needs deliberate topic settings, access permissions, secure client connections, and enough resources for the actual workload. Set a distinct state.dir for each instance sharing a machine, and provide suitable local storage for stateful processing. Adding application instances does not fix unavailable brokers, insufficient topic retention, or an overloaded output destination.

Before changing a running topology, check whether the new code can reuse the application's existing state and internal resources. A compatible change may retain the same identity and progress; an incompatible change may need a planned migration or rebuild. Treat the processing plan and its stored data as parts of the same deployed application.

Kafka Streams runs processing logic inside your application and manages the Kafka clients that execute it. A topology defines the record flow, while Serdes define how keys and values cross byte boundaries.

The application ID connects cooperating instances to shared processing progress. Tasks divide the work by partitions, and stream threads execute those tasks across the available instances.

Reliable operation also requires clear input contracts, compatible deployments, and deliberate recovery behavior. Kafka Streams can coordinate processing and restore supported state, while external side effects and business-level duplicate handling remain application responsibilities.