An online store runs an hourly job to calculate how many orders each store location has received. The report is useful for reviewing the day, but an operations team watching a sale needs fresher numbers. If one location stops receiving orders at 10:05, waiting until 11:00 to see the change is too long.

Stream processing lets an application update results as new events arrive. For the store, each order can contribute to a running count without waiting for the next hourly job.

This chapter explains how continuous processing works, how it differs from batch processing, and where Kafka fits. We'll follow an order-counting application to understand the role of state, time, and recovery in producing useful results.

An event stream is a sequence of events that producers publish over time. The store's orders.placed topic receives an event whenever the order service accepts a new order. More events can arrive tomorrow, next week, and for as long as the store operates.

We call this input unbounded because it has no predetermined end. That describes the logical stream, not an instruction to keep every record in Kafka forever. Retention still determines which records remain available.

Stream processing applies computations to this continuing input, producing new records or updating stored results as processing progresses. A processor might select orders from one country, calculate a count per location, or detect a pattern that should trigger an alert.

The application does useful work before the input is complete. After reading an order for store-17, it can update that store's count and continue waiting for more orders. A quiet period doesn't mean the calculation has finished.

The calculation can also run over retained history. If the application was offline for an hour, it can process the backlog and then continue with newly arriving records. The processing model is the same even though the results are temporarily behind current activity.

Continuous processing does not promise an instantaneous result. Publishing, fetching records, calculating results, and updating a dashboard all take time. A useful requirement is specific, such as making accepted orders visible on the dashboard within five seconds under normal load. That is a target for the whole pipeline to meet and measure.

A batch job processes a bounded set of input, such as all orders in an hourly export. It calculates a result for that set and finishes. Another run handles the next set.

The hourly job is a reasonable design when the business only needs an hourly report. Reducing the interval to one minute may also be sufficient. The choice depends on how fresh the answer needs to be and how much work each run performs.

With stream processing, a running application incorporates incoming orders into its results incrementally. To update a count, it can add one to the count it already has instead of repeatedly scanning all orders in the interval.

The diagram compares two ways to use the same order activity. Each path has a different point at which it begins calculating results.

Both paths can calculate order counts. The streaming path keeps an answer up to date while events continue arriving; the batch path produces an answer for each selected input set.

The distinction is about how the computation handles input. Kafka clients can fetch and send batches of records for efficiency while the application still performs stream processing. Some processing systems also use micro-batches, small groups that the system processes at short intervals, to handle a continuing stream.

A system can use both approaches. The store might maintain a live operations dashboard throughout the day and run a nightly reconciliation job to check totals against authoritative order data.

Kafka brokers store and serve records. The processing application reads those records and runs the business logic. Creating an orders.placed topic does not cause Kafka to calculate order counts.

For our dashboard, the order service publishes events, an application calculates counts, and a separate consumer writes the results into a database that the dashboard can query.

Here is the flow. The arrows show record movement; the applications reading from Kafka request the records through consumers.

The counting application acts as both a consumer and a producer. Its output is another stream that applications can read independently. An alerting application could consume the count updates without adding alert delivery to the counting logic.

Kafka Streams is a client library for building these processing applications. It runs inside your application process, separately from the brokers. It provides operations for transforming records and maintaining results across records, along with support for recovering processing state.

You can also implement stream processing with ordinary Kafka consumers and producers. A program that reads an order, transforms it, and publishes the result is already performing stream processing. A processing library becomes useful as the application needs to manage calculations across many records and recover them correctly after failures.

Kafka Connect has a different primary purpose: moving data between Kafka and external systems. A sink connector might write the count updates into the dashboard database if it supports the required update behavior. The counting application remains responsible for calculating those totals.

Some computations need only the current record. Selecting orders whose currency is USD, removing fields from a value, or converting a timestamp into a standard representation are examples of stateless processing. The operation doesn't need to remember earlier records to calculate the current result.

Counting orders requires memory. When a new order arrives for store-17, the processor needs that store's previous count. This is stateful processing, and state is the information the application retains between records.

Consider an event in orders.placed with the record key ord-1042 and this JSON value:

The event means the order service accepted this order. Our dashboard counts accepted orders, including orders that a customer or service might later cancel. A count of currently active orders would require additional events and different logic.

Suppose the processor groups accepted orders by store and one-minute interval. For now, assume all events arrive promptly and each event contributes exactly once. The count for store-17 in the interval starting at 10:00 begins at zero.

The application maintains a separate count for each store and interval. An order for store-22 leaves the count for store-17 unchanged.

The input key identifies an order, but the calculation groups by store. In a distributed processor, related records must reach the processing task responsible for that store's count. This may require repartitioning, redistributing records according to a new key. Merely including storeId in the JSON value does not place every order for that store in the same input partition.

Stateful work also includes joining related data. For example, matching an order with a later payment event requires retaining information until the counterpart arrives. Even if a lookup table lives in a database, a calculation that depends on that table depends on state.

State gives the application memory, but that memory needs a lifecycle. The application eventually needs to remove or archive counts for old intervals, and active counts need to survive a process failure. A continuously running application cannot keep every temporary value forever.

The phrase "orders per minute" needs a precise definition. The order service may accept an order at 10:00:58, but its event may reach the processor at 10:01:07. Which minute should include it?

Event time is when the event happened, according to the event's timestamp. Processing time is when the application processes it. For a dashboard showing when customers placed orders, we choose event time and use occurredAt as the source of that time.

The application must implement that choice explicitly. A JSON field named occurredAt does not automatically control a processing library's time calculations. The application must arrange for the processor to use the intended timestamp, and the producer must supply a trustworthy value.

A window groups records into a defined interval for a calculation. Our example uses fixed, non-overlapping one-minute windows in UTC. The 10:00 window includes events at or after 10:00:00 and before 10:01:00. An event at exactly 10:01:00 belongs to the next window.

The diagram shows why arrival time can change which result needs updating. The order reaches processing during the next minute, but its occurrence time belongs to the earlier window.

If the processor accepts this late event, it revises the 10:00 count. Kafka's partition ordering does not remove this issue: append order within a partition can differ from the order in which events happened.

The application needs a policy for how long earlier windows remain open to late records. Keeping them open longer can include more delayed data, but requires retaining state longer and postpones final results. Once the processor closes a window, the application needs a defined way to handle later arrivals, such as excluding them from live counts and reconciling them separately.

The mechanism for closing windows depends on the processing framework and its time model. A window's end timestamp alone does not guarantee that the processor emits a final result at that exact wall-clock time.

Window size and update frequency are separate choices too. A one-minute window can publish changing counts while the minute is still in progress. The processor can also buffer updates or wait until the window closes. For our operations dashboard, early updates are useful, provided readers understand that recent counts may still change.

After processing the first two orders for store-17, the count is 2. Suppose the application publishes the current total to analytics.order-counts, using store-17|2026-09-12T10:00:00Z as the record key and this value:

If another accepted event belongs to that window, the application can publish orderCount: 3 under the same key. The new record describes a revised total for the same store and interval.

The dashboard updater must replace the stored count for that key. Adding 2 and then 3 would produce 5, even though the correct total is 3. A stream of totals has different meaning from a stream of increments, and the event contract must make that distinction clear.

Kafka appends the new record; it does not edit the earlier count record in place. The dashboard database maintains the latest result that its updater has applied. If the topic uses log compaction, Kafka may eventually remove older values for a key, but readers must still understand update semantics.

Replacing a count with the same value also makes a repeated write easier to handle safely. The updater still needs to preserve update order for a key, or use a version check, so that a delayed retry of count 2 cannot overwrite count 3.

This is how a continuing stream can support a queryable view. The processor maintains the calculation, and a destination exposes the current answer to users.

The earlier count walkthrough assumed that each event contributes once. Failures make that something the application must ensure.

Suppose a processor reads partition 0 at offset 42, increments a count, and crashes before saving its progress. If recovery resumes at offset 42 while keeping the increment, the event contributes twice. If progress advances to offset 43 but recovery loses the increment, the recovered count is too low.

Correct recovery therefore connects input progress, state changes, and published results. Committing an offset by itself does not save a count or verify an output. After a failure or task reassignment, the replacement needs state that matches its restart position.

Kafka Streams supports an exactly-once processing mode that coordinates consumed offsets, Kafka output records, and recoverable state updates. That scope does not automatically include a separate database write or an HTTP request. The dashboard updater still needs its own safe write and recovery behavior.

Duplicate business events are another concern. If evt-7001 appears as two distinct input records, processing each record exactly once can still count the order twice. When duplicates are possible, the application needs a rule for recognizing them, such as tracking event IDs for a defined period. That tracking is itself state, and its retention determines how far apart duplicates can arrive while the application can still recognize them.

Replay needs similar care. Rebuilding a report by reading old orders into an already populated count would add those orders again. A rebuild can start with fresh state and a separate output destination, then replace the old view after validation. The required input history must still exist, or be recoverable from another source.

Even without failures, processing can fall behind. If orders arrive faster than the application can handle them, unread records accumulate and the dashboard becomes stale. Monitor how far processing has progressed and how old the data behind visible results is. A running process alone does not prove the dashboard is current.

Stream processing is useful when a fresh result changes what someone or something can do. An operations team can investigate a sudden drop in orders during a sale. A security application can flag repeated failed sign-ins while an attack is still happening. Both depend on receiving results soon enough to act.

The store's live dashboard benefits from incremental counts, but an accounting report that the team delivers the next morning may be simpler to produce with a batch job. That report may also need payment settlements, cancellations, and refunds that arrive well after the order service accepts an order. Faster order counts alone would not satisfy its requirements.

Some decisions require an immediate response before an operation completes. If checkout must verify stock before accepting an order, a processor reacting to OrderPlaced runs after that acceptance. The stock check needs to happen in the request path or in a workflow that waits for the required decision.

Start by defining the result and its acceptable delay. For our dashboard, that means specifying accepted orders per store and minute, choosing occurrence time, and deciding how to handle corrections. Those choices explain both why continuous processing helps and what the application needs to operate correctly.

Stream processing computes results over a continuing input, updating them as processing progresses. Kafka stores and distributes the records; processing applications apply the calculations.

Simple transformations can work on each record independently. Counts and other calculations across records need state, while time-based results need explicit window and late-data rules. Output contracts must explain whether records carry individual changes or replacement totals.

A useful streaming application combines timely results with correct recovery. Its input progress, state, and outputs must stay consistent, and downstream systems must handle repeated or revised results safely.