Applications produce events all the time: a customer searches for a product, stock levels change, or a delivery vehicle reports its location. Several applications may need the same events, each for a different purpose and at its own pace.

Kafka gives these applications a shared stream of events they can read now or return to later.

In this chapter, we’ll explore common uses for Kafka, what it contributes to each one, and what the applications around it still need to handle.

In an event-driven system, applications publish facts about what happened, and other applications react. An order service might publish OrderPlaced. Fulfillment can then prepare the order, while a notification service sends a confirmation.

This becomes especially useful as more applications need the events. You can add a reporting consumer without adding another call to the order service. Each consumer tracks its own progress.

For example, the order service writes a record to orders.placed with the key ord-1042. The value describes the accepted order, including an event ID, when it happened, and the details consumers need. Fulfillment, notifications, and reporting each use a separate consumer group.

The diagram shows the same event reaching all three applications. The arrows show data flow; each consumer requests records from Kafka.

If reporting goes down, fulfillment can keep working. Reporting can catch up later as long as Kafka still has the records it needs. The applications work at their own pace, but they still need to agree on what the events mean and their format.

The workflow also needs to account for unfinished or failed steps. Accepting an order does not mean fulfillment has reserved the stock. Fulfillment might later publish InventoryReserved or InventoryReservationFailed. The application decides what to do next, such as cancel the order or contact the customer. Kafka carries the events.

The order service also needs a reliable way to publish. Saving an order in a database and writing an event to Kafka are separate operations. A transactional outbox helps connect them: save the order and an outgoing event in one database transaction, then let a separate process publish the event. This keeps a durable record of what the publisher still needs to send. Publishers and consumers still need to handle retries and duplicates.

This pattern works well when the original request can finish without waiting for every reaction. If a purchase requires an immediate stock check before the store can accept it, the application still needs a way to get that answer in time.

A product catalog might store prices and descriptions in a database, use a search index for product searches, and send data to a warehouse for analysis. When a product changes, the search index and warehouse need the update too.

Change data capture, or CDC, captures inserts, updates, and deletes from a database. A connector that supports that database can send the changes to Kafka. For example, Debezium’s PostgreSQL connector can first take a snapshot of existing data, then stream changes from PostgreSQL’s write-ahead log, which the database uses for recovery.

The connector captures the changes, Kafka stores them in a stream, and consumers apply them to the search index and warehouse.

Each destination can keep its own pace. If the warehouse slows down, search updates can continue. To rebuild a destination, its consumer can start with a suitable snapshot and apply the changes that are still available. The source application does not have to resend every update itself.

Kafka Connect runs connectors that move data into and out of Kafka. A source connector brings data in; a sink connector writes it to a destination. Each connector has its own supported databases, formats, and rules for handling deletes and delivery.

For product prod-318, an update should target that same product every time a consumer processes it. Repeating the update should not create another product. Deletes need attention too, or a removed product could remain in search results.

These copies are eventually consistent: they can lag behind the source and catch up as consumers apply changes. Saving a new price in the database does not mean search results show it immediately. The application needs to account for that delay.

A database change may also tell you less than a business event. A new value in an order’s status column does not necessarily explain why it changed. CDC is useful for keeping data in sync, while an event such as OrderCancelled can give other services a clearer, more stable message to work with.

A store may want to track how customers move from searching to viewing products and placing orders. If checkout breaks during the day, a nightly report comes too late to help the customers affected now.

The application backend or an event collector can send activity events to Kafka. A processing application uses them to update results as they arrive. This is stream processing: turning incoming records into new records or updated state.

For example, a collector could send this JSON value to web.product-views, using the session ID as the key:

The session key helps determine how the producer divides input records among partitions. To count views per product, the processing application still needs to group the events by product.

Here’s one way to count views for each product in one-minute intervals. We’ve chosen one minute for this example; Kafka does not choose it for us.

Kafka stores the input events and output counts. The processing application does the counting, and the dashboard queries its own database. You could build the processor with Kafka Streams, a library that runs inside your application. The brokers themselves do not calculate the totals.

A window is a time interval a processor uses to group events. The processor needs to choose which time to use: when the event happened or when the processor handled it. If a view happened at 09:30:12 but arrived at 09:32:00, counting by occurrence time puts it in the 09:30 window. The application must decide how late an event can arrive and whether to revise earlier results.

Retries can inflate the count too. Receiving view-8201 twice may mean the collector sent the same view again, not that the customer viewed the product twice. The application needs to handle duplicates; Kafka does not automatically deduplicate records by their event ID.

The same pattern can detect a burst of failed sign-ins or unusual purchase activity. A processor identifies the pattern and sends an event to an alerting application. Your application defines the rules, handles false alarms, and decides how to respond. If the goal is to block a request before it completes, an alert the processor sends afterward will not be enough.

Logs describe individual operations. Metrics summarize behavior, such as request counts and error rates. As a system grows, several tools may need this data for search, alerts, and long-term storage.

Kafka can sit between the collectors and these tools. For example, a collector sends structured checkout error logs to ops.checkout-logs. One consumer makes them searchable, another looks for repeated failures, and a third archives batches to object storage.

The diagram shows three consumers reading the same log stream. Metrics can follow a similar path through a separate topic and processors suited to that data.

Each consumer uses its own group, so a slow archive does not hold up search or failure detection. Kafka can also buffer records while a destination is unavailable, within its capacity and retention limits. The destination tools still provide search, metrics queries, and alert rules.

Plan for the moments when you need logs most. A failing service may produce far more logs than usual, causing a backlog if consumers fall behind. If Kafka cannot accept more records, collectors need a fallback, such as a limited local buffer or a rule for dropping less useful records. Every buffer has a limit.

Choose log fields carefully. Request and order IDs can help trace a checkout problem without including payment credentials or full request bodies. If sensitive data enters a shared stream, it can spread to several destinations and become harder to remove.

For a small system with one monitoring tool, sending data directly may be enough. Kafka becomes useful when you need buffering, several independent readers, or the ability to process records again.

Devices send measurements over time: a delivery vehicle reports its location, or a refrigerated container reports its temperature. Dispatch, maintenance, and analysis applications may all need the same readings.

A gateway can receive and validate device messages, then send them to fleet.vehicle-locations. Devices do not have to connect to Kafka directly. The gateway can handle their protocols, authentication, and intermittent connections.

A record for vehicle-27 could use the vehicle ID as its key, with coordinates, a measurement time, and a device sequence number in the value. If the partitioning rule and partition mapping stay consistent, the vehicle’s records go to the same partition. Kafka keeps the order in which it appended the records. That order may differ from the order in which the device took the measurements.

For example, after leaving a tunnel, a vehicle might upload buffered readings after a newer reading has already arrived. The live location consumer should avoid replacing the current location with an older one. It needs a rule for deciding which reading is newer, using reliable timestamps, sequence numbers, or both.

Delayed readings can still help reconstruct a route or calculate travel times. The diagram shows two consumers using the same readings for different purposes.

The consumers use separate groups. One can add an older reading to route history while the other keeps the newer live location. Each application decides how to use the timing information.

This setup still depends on the quality and timing of device data. One noisy device can use a large share of capacity, and a disconnected device cannot provide live updates. Kafka distributes the readings it receives, but cannot make devices report on time. A safety-critical response at the device needs to work even when the remote pipeline is unreachable.

In event sourcing, recorded business events are the source of truth. The application builds its current state from that history. Events such as OrderPlaced, OrderCancelled, and OrderShipped describe what happened to an order over time.

If an orders table is the source of truth and the application publishes events after updating it, that is a different design. Sending those events through Kafka does not make it event sourcing.

Kafka can store the event history and feed derived views: data that applications build from events to answer specific questions. A support application might build an order timeline, while reporting calculates daily totals. You can rebuild a view if you still have the events it needs and processing code that can interpret them.

Here, the application checks whether a requested change is valid before recording an event. Separate consumers use the history to build their views.

Each view builder uses its own consumer group. Each builder can rebuild its view from the required history, which remains the source of truth. The application handles validation, including changes that clients request at the same time.

Retention must support the history you need. If a topic deletes old events, you cannot rebuild state that depends on them unless you have another complete copy or a suitable snapshot plus later events. Log compaction can remove older records for a key while keeping newer state, so it does not preserve every event in an order’s lifecycle either.

Concurrent changes need care too. If two requests try to cancel and ship the same order at the same time, partition ordering alone will not reject the conflict. The application must decide which change is valid and allow only the appropriate writes.

Event sourcing is a choice about how your application stores and manages state. If you only need an activity history, a database history table and published events may be enough. Keeping events in Kafka also does not make them a tamper-proof audit archive. Access controls, deletion rules, and archival storage need their own design.

Across these examples, Kafka provides the same core benefit: several applications can read a shared stream at their own pace and return to records Kafka still holds. Each use case also brings its own design questions.

Before choosing Kafka, identify who produces the data, who needs to read it, how much delay is acceptable, and how far back readers need to replay. Decide what happens when a consumer goes down and what processing the same record twice would do.

Publishing a record does not mean a consumer has updated a search index, changed a dashboard, or fulfilled an order. A committed offset records a consumer’s reported progress, but does not verify that business result. Your application needs to connect saved progress to completed work.

For a small workload with one reader and no need for history, an API, scheduled transfer, or task queue may be simpler to run. Kafka is a good fit when keeping a shared stream and letting applications process it independently solves a problem you actually have.

Kafka can connect services, distribute database changes, feed live analytics, collect operational data, share device readings, and support event-sourced applications. It stores the shared records; connectors capture data, processors transform it, and destination applications put it to use.

For each use case, decide how to publish reliably, how long to keep records, and how to handle ordering, duplicates, and delays. Those choices turn a shared event stream into a system that works for your application.