Imagine an online store application. When a customer places an order, several parts of the system may need to react. One service sends a confirmation email, another updates analytics, and another starts the fulfillment process. These applications may run at different speeds, and some may be temporarily unavailable.

Apache Kafka helps them communicate through streams of events. Applications can publish events to Kafka, Kafka stores them, and other applications can read those events when they are ready.

In this chapter, we’ll look at how this works, why storing events is useful, and how multiple applications can consume the same event independently.

An event is a record of something that happened. In an online store, OrderPlaced means the store accepted an order. It does not mean that the order has shipped or that every application has finished reacting to it.

An event should include enough information for consumers to understand what happened without having to guess. For example, an order event might use a JSON payload like this:

Here, the event expresses totalMinorUnits in cents, so 129900 represents $1,299.00. Including both the unit and the currency makes the amount unambiguous.

The eventId identifies this specific event, while orderId identifies the order the event is about. If the customer or store later cancels the order, publish a new cancellation event with a different eventId and the same orderId.

An event stream is a sequence of events that applications keep producing. As customers place more orders, the system keeps generating new OrderPlaced events. Other applications can process those events as they arrive instead of waiting for a scheduled batch or export.

This is the idea behind event streaming: continuously capturing events, making them available, and processing new events as applications produce them.

Continuous does not necessarily mean instantaneous. An analytics dashboard, for example, may lag slightly behind the order service because of network latency, processing time, or a temporary outage.

Apache Kafka is an open-source, distributed event streaming platform. Distributed means that Kafka can spread its data and workload across multiple servers that work together as a cluster.

Kafka gives applications a shared place to publish events and allows other applications to read them independently. It also stores those events for a period of time, so consumers can process them later or replay data Kafka still holds.

Consider an order service that directly calls a notification service and an analytics service whenever it accepts an order. The order service now has to handle failures for both integrations. As more applications need order data, it must keep adding more calls, retries, and failure-handling logic.

With Kafka, the order service can publish a single order event to a shared stream. Any interested application can then read that event independently.

The order service no longer needs to know about every application that uses its events. If you add a reporting application later, it can start reading the existing stream without requiring another integration in the order service, as long as the events contain the data it needs.

This reduces direct dependencies between applications, but it does not remove all coordination. Producers and consumers still need to agree on the event contract, including what OrderPlaced means and the format of its fields.

The producer must also publish events reliably. Saving an order to a database and publishing an event to Kafka are normally two separate operations. Kafka does not automatically make both operations succeed or fail together. The application needs a strategy to keep them consistent.

A producer is an application or component that writes records to Kafka, while a consumer reads them. In our store example, the order service is the producer, and the notification and analytics applications are consumers.

Kafka organizes records into named topics. For example, the order service might publish OrderPlaced events to a topic called orders.placed.

A record is the unit of data Kafka stores. Its value could contain the JSON event we saw earlier, while its key could be ord-1042. Kafka stores the key separately from the value. The key can influence which partition receives the record. Records also include a timestamp and can carry headers for additional metadata. JSON is only one possible format; Kafka does not require it.

Kafka runs on servers called brokers. Brokers store records and handle requests from producers and consumers through Kafka client libraries. Consumers pull records from Kafka rather than Kafka calling application endpoints.

If both the notification and analytics applications need to process every order event independently, they should use separate consumer groups. A consumer group contains consumers that share the work of reading its subscribed topics. Each group tracks its progress independently.

In this example, order-notifications and order-analytics are separate consumer groups, with one consumer in each group for simplicity.

Reading a record does not remove it from Kafka. The notification application can process an event without preventing the analytics application from reading the same event later.

A topic contains one or more partitions. Each partition is an ordered log: Kafka appends new records to its end and assigns each record an offset, a position within that partition.

The example below shows one partition of orders.placed. The first record carries the event for ord-1042; the next two describe different orders. Assume both applications process records sequentially and have saved their progress after completing their work.

Notifications has finished offsets 0 through 2 and is ready for offset 3, which has not arrived yet. Analytics has finished offset 0 and still needs offsets 1 and 2. Both read the same stored records, but their progress is independent.

Committing an offset saves a group's restart position in Kafka. The committed offset identifies the next record to read. It is a checkpoint the consumer reports, not proof that Kafka has verified an email delivery or a database update.

Offsets belong to individual partitions. Another partition can also have an offset 0. Kafka preserves record order within a partition; it does not provide one total order across every partition in a topic. The single partition here makes the example readable and is not a production sizing recommendation.

Storage has limits. A topic's cleanup policy determines which records remain available. For this example, assume a policy that removes old data according to configured time or size limits. Reading a record, or committing an offset past it, does not trigger that removal. A slow consumer can therefore lose access to unread history if cleanup removes it before the consumer catches up.

Suppose the analytics service goes down while notifications keeps running. Kafka can continue storing new order events while analytics is offline, as long as Kafka itself is available and has enough capacity.

When analytics comes back, it can continue from its last saved offset and process the events it missed, as long as Kafka still holds those events.

Kafka also lets consumers intentionally read old events again, a process we call replay.

For example, imagine a bug caused the analytics service to leave some orders out of a report. After fixing the bug, the team can start a separate consumer from an earlier offset and rebuild the report using the original events.

Because the rebuild uses a separate consumer group, it can reread the events without affecting the notification service or any other consumer's progress.

Replay only works while the events still exist in Kafka. Once Kafka removes them under the topic's retention policy, consumers can no longer replay them from Kafka.

There is also an important catch: processing the same event more than once can cause problems. For example, a consumer might send a confirmation email and then crash before saving its offset. After restarting, it may read the same event again and send the email twice.

Consumers therefore need to handle duplicate processing safely. One common approach is for each consumer to store a unique event ID and check whether it has already processed that event before performing the action again.

The key idea is that publishing an event, reading it, saving an offset, and completing the actual business action are separate steps. A successful write to Kafka only means Kafka stored the event. It does not mean every consumer has already finished processing it.

Kafka is useful because it stores events in one place while allowing multiple applications to read and process them independently. This works well when several services need the same data, when consumers may temporarily go offline and catch up later, or when you need to replay past events.

Kafka also scales well. Topics contain partitions, so Kafka can spread data and processing across brokers. Kafka can also replicate partitions to keep copies on multiple brokers. This helps Kafka continue operating when a broker fails and reduces the risk of data loss.

However, simply using Kafka does not make every message perfectly safe. Durability depends on factors such as the replication factor, producer acknowledgment settings, and the type of failure that occurs.

Kafka also adds operational responsibility. You need to manage storage, capacity, security, retention, and consumers that fall behind. Managed Kafka services can handle much of the infrastructure, but your application still needs to define events clearly and process them correctly.

Kafka is also not the right tool for every problem. If one service simply needs an immediate response from another, a direct API call may be enough. And if a small application only needs to run a simple background task, adding Kafka may introduce more complexity than value.

Kafka makes the most sense when you have a real need for shared, durable event streams that multiple applications can consume independently.

Kafka lets producers write records to topics and lets consumers read the stored data independently. Each topic consists of partitions, and each partition has its own record order and offsets.

Retention makes recovery and replay possible while the required records remain available. Separate consumer groups can use the same history for different purposes. Applications remain responsible for publishing reliably, tracking progress correctly, and making repeated processing safe.