An order service publishes an event, a reporting application reads it, and a fulfillment service reads it a few seconds later. Tomorrow, another application may need the same event to rebuild a view of yesterday's orders. Kafka supports these uses by keeping records in a shared log, separate from the progress of any reader.

That storage model explains much of Kafka's behavior: how it preserves order, why reading leaves records available, and how replicas help the system recover. In this chapter, we'll follow an order's history through a partition log and distinguish storing a record from committing it and processing it.

Our examples use a KRaft-based cluster, regular consumer groups, and non-transactional records. We'll assume consumers fetch from partition leaders and the example's records remain within retention.

A log is an ordered sequence of records that grows by appending new records to its end. A commit log gives a system an accepted sequence of entries it can use during normal operation and recovery.

Databases use related structures, often called write-ahead logs, to record changes before applying them to their main data structures. After a failure, the database can use its log to recover. Kafka exposes a retained log to applications, which can read its records and build their own state from them.

For example, a reporting application can turn order events into a table of current order statuses. A fulfillment application can use the same events to decide which orders are ready to ship. Each application interprets the records for its own purpose.

Kafka does not check whether an OrderPaid event corresponds to a real payment. It stores the bytes the producer supplies. The producer is responsible for publishing a meaningful event, and each consumer is responsible for applying it correctly.

There is also a boundary between Kafka and the order service's database. Saving an order row and publishing an event are separate operations unless the application uses a design that coordinates them. Calling Kafka a commit log does not make those operations atomic.

Suppose an orders.events topic contains changes to an order throughout its lifetime. The producer uses the order ID as the record key and consistently routes events for ord-1042 to partition 0.

Here is a short portion of that partition. The table shows selected event fields, and each event has its own ID.

The offset identifies a record's logical position within the partition. It is not a byte address or an order ID. Other orders can appear between these events in a real log; the example keeps the three together so their progression is easy to follow.

Publishing OrderPaid appends another entry. It does not replace OrderPlaced, even though both records have the same key. Likewise, a correction would arrive as a new record describing the correction.

The diagram shows the append order. Its arrows indicate increasing offsets, not requests between services.

Once Kafka commits and retains these records, readers encounter the same entries in the same partition order. A reporting consumer can apply OrderPlaced, then OrderPaid, then OrderShipped to derive the order's current status.

Append-only describes how applications write. It does not mean the underlying files never change or that every record remains forever. Kafka can remove old data through cleanup, and recovery can truncate an uncommitted tail that conflicts with the elected leader's log. Applications cannot edit an existing record in place.

Kafka distributes logs in two ways. Partitioning divides a topic into separate ordered sequences. Replication keeps copies of each sequence on different brokers.

The distinction matters because each partition is its own log. Partition 1 can have an offset 120 unrelated to offset 120 in partition 0. To locate an entry, you need the topic, partition, and offset together.

Kafka does not define a total order across those partitions. If a payment event is in partition 0 and a shipping event is in partition 1, comparing their offsets tells you nothing about which happened first. Timestamps also do not create a shared append order.

For ord-1042, our example assumes the producer sends the events in the intended order and keeps them in partition 0. Kafka preserves the resulting log order. It does not rearrange events into business order if the producer publishes them incorrectly, and consumers must preserve any required order when applying them.

Now give partition 0 three replicas. Broker 1 leads it, while Brokers 2 and 3 follow. The leader accepts writes, assigns offsets, and appends records. Followers fetch from the leader and append the same records at the same offsets in their own logs.

These replicas are copies of one logical sequence. They do not give the order three independent positions or create three events for consumers to process. Other partitions can have different leaders, spreading the write workload across brokers.

KRaft controllers manage cluster metadata and partition leadership. Their metadata log is separate from orders.events; brokers store the order records in their partition replicas.

A leader can have a record that its followers have not fetched yet. That creates a difference between the leader's local log and the portion Kafka has committed through replication.

Two positions help describe this difference:

Suppose all three replicas remain in the in-sync replica set, or ISR: the replicas Kafka considers sufficiently caught up, including the leader. They have all copied records through 121, and the leader has advanced its high watermark to 122. The leader then appends OrderShipped at 122, but the followers have not fetched it yet.

This snapshot shows both positions. We assume the topic meets its minimum in-sync replica requirement throughout the example.

Being in sync does not mean a follower has zero lag at every instant. Both followers can still belong to the ISR while briefly missing the latest append.

After they copy 122 and the leader learns of their progress, it can advance the high watermark to 123. The record at 122 then becomes available to consumers in this non-transactional example. Consumers do not read the leader's uncommitted tail simply because it exists in a local file.

Producer Acknowledgments and Durability

The producer's acks setting controls when it receives confirmation. With acks=1, the leader can acknowledge 122 after its local append, before followers copy it. If the leader fails at that point, a replacement may not contain that record.

With acks=all, successful acknowledgment waits for the required replication. If all three replicas remain in the ISR, all three must have the record. Setting min.insync.replicas=2 does not reduce that wait to any two replicas; it sets a minimum ISR size for successful writes using acks=all.

A missing acknowledgment leaves the producer uncertain: the record may have committed before the producer lost the response. Retrying can therefore repeat a write. Producer idempotence handles supported client retries, but an application submitting the same business event again as a new send can still create a duplicate. An offset identifies a log entry, not a unique business action.

If Broker 1 fails after Kafka commits 122 and Broker 2 takes over with that record, consumers can continue using the same partition offsets. A returning replica may need to discard conflicting uncommitted entries before following the new leader.

Durability still depends on surviving copies and safe leader selection. Losing every usable copy, or allowing a stale replica to become leader through unclean election, can lose data. Replication cannot protect against every possible failure.

An acknowledgment also does not normally mean each replica has forced the record onto physical storage with fsync. Filesystem writes can remain in the operating system's page cache before the operating system flushes them. Kafka relies heavily on replication for durability, so simultaneous failures that lose unflushed data across replicas require separate consideration from one broker process failing.

The Different Meanings of Commit

Kafka uses the word commit in several contexts. Keeping them separate prevents a successful operation in one layer from looking like completion in another.

Transactional reads add another visibility boundary. A consumer using read_committed waits for transaction outcomes and filters aborted transactional records, so it may not read all the way to the high watermark. Even read_uncommitted consumers still cannot read beyond the replication-committed boundary; the setting refers to transaction isolation, not permission to read an unreplicated tail.

Once Kafka commits records through 122, reporting and fulfillment can read the same history at different speeds. Kafka keeps their progress separate from the order records.

Suppose reporting has processed through 121 and committed 122, while fulfillment has processed through 122 and committed 123. These commits identify the next restart positions. They are not markers Kafka writes into the order events themselves.

The diagram shows one retained log serving both groups. Each arrow describes where that group would resume using its saved progress.

Reporting still needs to apply OrderShipped. Fulfillment has reached the current end and waits for more records. Neither group's progress removes 120, 121, or 122 from the log, and adding a third group does not create another stored copy of the topic.

A consumer's current position can be ahead of its committed offset because fetching, processing, and saving progress happen separately. Suppose reporting updates its database for 122 and crashes before committing 123. Its replacement can resume at 122 and repeat the update. The application needs a way to handle that repetition safely, such as recording the event ID alongside the database change in one database transaction.

Replica recovery preserves access to the log. Consumer recovery uses that log to resume application work. A successful recovery of either one does not automatically complete the other.

Separating records from reader progress lets an application revisit earlier input. Suppose a reporting bug left shipped orders marked as paid. A rebuild process can start at 120, apply the three events with corrected logic, and write ord-1042 as shipped in a replacement table. The live reporting group can continue with its own offsets.

That table is derived state: a view the application calculates from processed events. The same retained sequence can support several such views without requiring the producer to publish again for each reader.

Replay needs enough input to reconstruct the desired state. Starting at 121 may be insufficient if the application needs the order total from 120. A larger rebuild needs all required history or an appropriate starting snapshot, plus code that can still interpret the events.

External dependencies matter too. Recomputing a historical total using today's product price can produce a different result. Events should carry the historical facts the calculation needs, or the application needs another way to retrieve them. Replaying fulfillment events also needs care: rebuilding a status table should not accidentally purchase another shipping label.

Retention places a practical limit on this capability. Kafka can delete old records even when a group has not read them. Log compaction can remove older values for a key, so a compacted log may support rebuilding current state without preserving every intermediate event. A saved offset cannot restore a removed record.

For the order history here, retaining all three events lets the application reconstruct the complete transition. If only a later event remains, what the application can rebuild depends on what that event contains. The decision to keep history, snapshots, or both follows from the recovery the application needs.

Kafka stores each partition as an ordered log and replicates that log across brokers. New records append at the end; the high watermark separates replication-committed records from the uncommitted tail. Ordering and offsets belong to individual partitions.

Consumers read retained records and save their progress independently. This supports separate applications, recovery, and replay, provided the required history remains available. Replication commitment, offset commits, and transaction commits describe different outcomes, and applications remain responsible for turning the stored events into correct business results.