A consumer stops after reading some order events. When it starts again, the consumer needs to know exactly where to resume. The same question appears when you rebuild a report, inspect a missing event, or check how far an application has fallen behind.

Offsets make those positions explicit. But a record’s offset, the end of a log, and a consumer’s saved progress mean different things. In this chapter, we’ll separate those meanings and use them to reason about reading, recovery, replay, and missing history.

An offset is a non-negative integer that identifies a record’s logical position within a Kafka partition. When the partition leader appends records, Kafka assigns their offsets. Producers select a destination partition, but they do not choose the offsets of new records.

For example, an OrderPlaced event might have this location:

The key and event ID belong to the application’s data. Offset 42 identifies where this particular record sits in partition 0. Another partition can also contain offset 42, so an offset alone is not enough to locate a record.

Within a cluster and a topic’s lifetime, the combination of topic, partition, and offset identifies a stored record. Deleting and recreating a topic with the same name creates a new history; old positions are not durable business identifiers for the replacement topic.

Offsets describe log order. A record at offset 42 precedes one at 43 in the same partition, even if its event timestamp is later. Offsets from different partitions do not establish an order between their events.

An offset is also not a byte address in a file. Records vary in size, and Kafka stores logs in segment files that Kafka can clean up or compact. Kafka uses indexes and log metadata to find the data corresponding to a logical offset.

To understand a position, first identify the range of history the partition currently retains.

The log start offset is the lower boundary of that retained history. It begins at 0 for a fresh partition and can advance as Kafka removes older history. The log end offset, often abbreviated LEO, is the next append position in a particular replica’s log. Here, we’ll use the leader’s log end.

Suppose partition 0 of orders.placed retains five records, at offsets 40 through 44. Kafka has removed earlier records. Assume Kafka has fully replicated these five records and made them available to read, with no transactions or gaps in this example.

The diagram shows the retained records and the boundary after them. Offset 42 is the record for ord-1042 described above.

The log end is 45, even though there is no record at offset 45 yet. It is an exclusive boundary: the retained records in this example have offsets greater than or equal to 40 and less than 45.

A consumer positioned at 45 is caught up with this log. It can wait for new records rather than reread 44. If the next append is an ordinary record, it receives offset 45, and the log end advances to 46.

An empty partition that has never received data has start and end offsets of 0. A partition whose older data Kafka has removed can be empty at a much larger position. Empty does not always mean that its offsets have returned to zero.

Cleanup does not renumber the remaining records. If the start advances from 40 to 43, the retained records still have offsets 43 and 44; they do not become 0 and 1.

Offsets are ordered, but the records visible to a consumer need not have consecutive offsets. A gap does not automatically indicate corruption or failed delivery.

One cause is log compaction, which removes superseded records for a key while preserving newer state. Consider a separate topic, orders.current-state, that uses compaction. Assume three records in one partition describe these states:

Once the older state at 41 is eligible for compaction, Kafka can remove it. The surviving records keep their original offsets. The diagram shows one possible cleanup result, not an immediate change on every write.

Here, 41 is still within the partition’s valid offset range, but compaction has removed its record. Reading from 41 proceeds to the next available record, at 42. If the log start later advances beyond 41, requesting 41 is an out-of-range situation. A missing record inside the valid range and a position outside that range are different cases.

Transactions can create other apparent gaps. Kafka has internal transaction-control records that ordinary consumers do not receive as application records. A consumer that reads only committed transactions also skips records from aborted transactions.

Subtracting the start offset from the end offset gives the size of the offset range. It may exceed the number of business records a consumer can read. Even when the arithmetic happens to match the count in a simple example, that equality is not a general guarantee.

A regular Kafka consumer group maintains progress separately from the stored records. Two positions matter:

The current consumer position is where that consumer will continue reading in the running process. In the Java consumer, it advances as poll() returns records to the application. Receiving a batch does not mean the application has finished processing it.

The committed offset is the group’s saved restart position for a partition. Committing means storing that checkpoint in Kafka. A replacement consumer normally resumes there after a failure or reassignment, assuming the checkpoint is still valid.

Both use the convention of identifying where to continue, not the last completed record. For our gap-free order log, finishing offset 41 and all earlier work allows the application to save 42 as its next position.

Now suppose order-reporting has committed 42. Its consumer polls and receives records 42, 43, and 44. The current position advances to 45, but the committed position remains 42 until another commit succeeds.

The diagram shows this snapshot immediately after the poll, before the application processes the returned batch.

The consumer has delivered all three records to the application, but the application has not processed them yet. If the process fails now, its replacement can start at the saved position 42 and read the batch again.

Suppose instead it applies the report update for 42, then fails before saving progress. Restarting at 42 can repeat that update. Conversely, saving 45 before completing the batch could cause a restart to skip unfinished work at 43 and 44.

The application’s completed work is therefore a third fact to track. The consumer knows which records it returned, and Kafka stores the checkpoint it received; it does not verify an external database update or email delivery.

After completing the entire batch and all earlier work, the application can commit 45. That commit does not delete records 42 through 44. It only changes where this group should resume.

Every partition has its own positions. A group can be caught up on partition 0 while still behind on partition 1, and another group can have different checkpoints for both.

Seeking changes a consumer’s current position in an assigned partition. Seeking backward allows it to reread retained history; seeking forward skips positions for that reader. Neither action changes the records stored in Kafka.

For example, seeking to 42 in our order log lets a consumer read 42, 43, and 44 again. Their offsets and values stay the same. This is replay. Republishing their contents would instead create new records at newly assigned offsets.

A seek does not itself change the group’s committed position. A later automatic or explicit commit can do so. For an independent replay, use a separate group or a reader with direct partition assignments and settings that prevent it from changing the live group’s checkpoint. The application must also handle repeated business actions safely.

Replay stops being possible when Kafka no longer retains the required history. Suppose the group saved 42, but cleanup later advances the log start to 43. The checkpoint still says 42; saving it did not reserve that record or protect it from cleanup.

For the Java consumer, auto.offset.reset determines the fallback when it has no initial position or its position is out of range. Three commonly used choices are:

These are not the only options Kafka 4.3 supports, but they illustrate the main recovery choices. No reset policy reconstructs removed records.

earliest means the earliest history still available, not necessarily offset 0. It also does not rewind a consumer every time it restarts. If the group has a valid committed position, the consumer normally resumes there instead of applying the reset fallback.

So far, our order-log example assumes every appended record is available to read. In a live replicated cluster, the leader can have records at the end of its log that consumers cannot read yet.

Three boundaries help distinguish storage from visibility:

A transaction can group Kafka writes into work the producer commits or aborts together. The LSO is the smaller of the high watermark and the first offset of an open transaction. read_committed consumers filter records from aborted transactions even after the stable boundary advances.

For example, suppose a leader’s LEO is 48, its high watermark is 46, and an open transaction begins at 44. The LSO is then 44. Records at 46 and 47 are beyond the ordinary read boundary, while a read_committed consumer must stay below 44 until transaction outcomes permit that boundary to advance.

The term “end offset” therefore needs context. In the Java consumer API, endOffsets() returns the high watermark for read_uncommitted consumers and the LSO for read_committed consumers. It does not always return the leader’s physical log end.

Despite its name, read_uncommitted does not expose the leader’s unreplicated tail. Its “uncommitted” refers to transaction outcomes. A transaction commit is also separate from a consumer group committing its restart offset.

Offset lag is a difference between an end position and a consumer-related position. To interpret a metric correctly, identify which end boundary and which consumer position it uses.

Return to the simple order log ending at 45. The group has committed 42, while the running consumer has polled through 44 and has current position 45:

Both differences are correct. The first says the saved restart point is three offset positions behind. The second says the running consumer has received everything up to this end boundary. Neither, by itself, says how much business processing has completed.

In this gap-free example, three offset positions correspond to three records. With compaction, transaction filtering, or internal control records, an offset difference can differ from the number of application records waiting for processing.

Lag is not a duration either. A gap of a thousand offsets could represent milliseconds of busy traffic or hours of sparse traffic. Record sizes and processing costs also vary, so the same offset gap can require very different recovery times.

When investigating a consumer, compare the retained start, the relevant end, the current position, and the committed position for the affected partition. If the checkpoint is below the retained start, history is missing. If the current position is ahead of the checkpoint, the consumer has read beyond what it has saved. If it has reached the readable end, additional waiting may be normal even while the leader has newer, not-yet-readable data.

An absent committed offset means there is no saved checkpoint available; do not treat it as a committed value of zero. And because the log and consumer can move while you inspect them, separate observations may reflect slightly different moments.

Offsets identify logical positions within a partition. The retained start marks the beginning of available history, while end boundaries describe how far Kafka has appended data or made it readable. Cleanup can remove records without renumbering the survivors, so offset ranges are not always record counts.

A consumer’s current position tracks reading, and its committed offset saves a restart point. Neither proves that application work is complete. Keeping these positions distinct makes it easier to reason about replay, recovery, missing history, and lag without confusing stored data with processed data.