A busy Kafka partition keeps receiving records while consumers read both recent and older data. Eventually, Kafka also needs to remove expired history. Keeping everything in one growing file would make that cleanup awkward: removing the beginning could require rewriting data that should stay.

Kafka divides each partition log into segments so it can manage portions of the history separately. We'll follow a segment as it receives records, stops accepting new appends, and becomes eligible for cleanup. Along the way, we'll see how segment boundaries affect reads and what smaller or larger segments change operationally.

Our examples use a KRaft-based cluster with local storage and no tiered storage. The orders.placed topic uses cleanup.policy=delete, and its records are non-transactional. Unless stated otherwise, the illustrated records remain available and have contiguous offsets.

A log segment is a portion of a partition's ordered log, which Kafka stores as a data file with supporting indexes. Its data file contains record batches. Kafka manages multiple segments together as one logical sequence of records.

At any moment, one segment in a local partition replica is the active segment. It receives normal appends. Older, inactive segments remain available for reads while Kafka continues writing to the active one.

Suppose partition 0 of orders.placed has the following layout. The arrows show increasing offsets within the same partition.

A consumer can read offset 75 from the first segment while the broker appends new records after 249 in the last one. Creating another segment does not create another partition or give a regular consumer group another unit of parallel assignment.

The segment's base offset identifies its starting position in the partition log. Kafka uses it in filenames, with leading zeros to make 20 decimal digits. For the segment beginning at 200, the main files are:

Other supporting files may also exist. The three files above belong to one segment; they are not three copies of its records.

The round offset ranges in the diagram are illustrative. Kafka rolls segments according to storage conditions, not every 100 records. Different record sizes, compression, and traffic rates produce different offset ranges in similarly sized files.

A segment roll switches normal appends from the current segment to a new one. The previous segment remains part of the partition's readable history.

Continue with the active segment beginning at 200, which contains complete batches through offset 249. The next accepted batch will contain offsets 250 through 254. Suppose appending it would exceed the segment's configured size limit.

Kafka rolls before appending that batch. In this ordinary leader-append example, the new segment begins at 250, and the whole batch goes into its data file. Kafka does not split a record batch between two segment files to fill the remaining space.

The diagram shows the change in the active segment. The diagram leaves out the older segments beginning at 0 and 100 because this roll does not change them.

The new data filename is 00000000000000000250.log, with corresponding indexes. After the append, the local log end offset is 255, the position after the last appended record. Offsets continue across the boundary; they do not restart at zero.

Rolling does not copy all the older records into the new file. Kafka directs new appends to the new segment and finishes managing the old segment’s indexes and other metadata. An inactive segment can still have open files or mapped indexes because Kafka may need to read it.

The words active and inactive describe appends, not record visibility. Consumers can read committed records from an active segment. Likewise, making a segment inactive does not establish replication commitment, complete a transaction, or save a consumer's progress.

Size and elapsed time are the main controls operators use for segment rolling. Kafka also rolls when supporting indexes or relative-offset representation reach their limits, so a segment can stop growing before its data file reaches the configured size.

The relevant topic settings are:

These topic settings are overrides; when absent, the corresponding broker defaults apply. They are not interchangeable with retention.bytes or retention.ms, which govern how much history Kafka keeps.

Size-Based Rolling

Kafka considers both the active segment's current data size and the bytes it is about to append. For example, suppose a segment has 63.5 MiB of data, its limit is 64 MiB, and the next valid batch is 0.75 MiB. The combined size would be 64.25 MiB, so Kafka rolls and places the batch in the new segment.

The old file finishes below 64 MiB. This is expected because batches stay intact. The limit also applies to the data file, not the total space its data, indexes, and other supporting files occupy.

Record size limits still matter. Kafka does not make an oversized batch acceptable by dividing it across segments. A size setting that cannot accommodate the application's valid batches can cause writes to fail.

Compression affects how quickly a segment fills because it changes the stored batch size. Two partitions receiving the same number of events can roll at different rates if their payload sizes or compression ratios differ.

Time-Based Rolling

A quiet partition may take a long time to fill a segment. A time-based roll condition lets Kafka finish a segment without waiting for its size limit, which helps make older data available for cleanup.

Do not interpret segment.ms as an exact timer that creates a new file at every interval. Kafka evaluates roll conditions during append processing, and retention processing can also force a roll when needed to remove eligible data. An idle partition need not accumulate an endless sequence of empty segments merely because time passes.

The timing also depends on Kafka's timestamp handling and when it evaluates the condition. A file's creation time alone is not enough to predict every roll precisely. Use the configured interval to reason about behavior, then check the partition's actual traffic and segment timestamps when investigating a specific case.

segment.jitter.ms spreads time-based rolls by subtracting a random amount from the interval. This can reduce the concentration of roll activity when many partitions would otherwise roll together. It is not a way to make records expire at a more precise time.

Consumers request offsets within partitions. They do not request particular segment files, and a roll does not require them to change their subscription or restart.

Suppose a consumer requests offset 248 after the roll in our example. Kafka locates the segment beginning at 200, uses its index to find a nearby file position, and reads the relevant batches. As the consumer advances to 250, Kafka serves records from the segment beginning at 250.

A fetch response or client buffer may cover only part of that progress. The application should not expect one poll result to correspond to one segment. Batch boundaries, fetch limits, and available data determine how records arrive at the client.

The broker's lookup starts with the segment boundaries, but those boundaries are not a count of the records still present. For example, compaction can remove entries without renumbering the remaining offsets. A segment beginning at 200 with the next segment beginning at 250 does not universally contain exactly 50 application records. Our delete-only, contiguous example does, by assumption.

Segment rolling also does not split an order's logical history. Records with key ord-1042 can appear in different segments of the same partition while retaining their partition order. Segments organize storage; the key and partitioning strategy determine where the producer routes related events.

Segments give Kafka a practical unit for deleting old data. Under a delete cleanup policy, Kafka can remove eligible old segments and their supporting files while keeping newer segments. It does not need to rewrite every retained record just to remove the beginning of the log.

Time-based retention evaluates a segment using its largest record timestamp. Consider an inactive segment containing records whose timestamps range from 10:00 to 10:20, with a one-hour retention period. Assume these timestamps reflect the intended event times and no other deletion condition applies.

Just after 11:00, the oldest records are more than one hour old, but the segment's largest timestamp is still less than one hour old. The whole segment is not yet eligible under the time condition. It passes that condition after 11:20, and actual removal follows Kafka's cleanup scheduling and safety checks.

This means records near the beginning of a segment can remain longer than the configured retention duration. Segment boundaries influence the granularity of cleanup, not just file organization.

The diagram shows the deletion lifecycle for one segment. These are conceptual stages, not names of Kafka protocol states.

Rolling alone does not make a segment eligible for deletion. Kafka also needs the relevant cleanup condition and replication safety conditions to hold. When retention needs to remove the active segment, Kafka can roll to preserve an active append destination before removing the old one.

Logical removal and physical file deletion are separate steps. Kafka delays file deletion, using the delay in file.delete.delay.ms, so a directory listing can briefly show files that are no longer part of the readable log. Their presence does not guarantee that consumers can fetch their records.

Consumer offset commits do not hold segments in place. If cleanup removes offsets 0 through 99, a group still expecting to resume at 75 cannot retrieve that history from this local-only topic. It must handle an unavailable starting offset according to its recovery policy.

Compaction uses segments differently: it rewrites eligible older data to discard superseded values for keys. Thus, an inactive segment receives no normal appends, but Kafka may change its physical files later. Cleanup and recovery can replace, remove, or truncate stored data while preserving the applicable logical guarantees.

Smaller segments create finer cleanup boundaries. They also create more data files, indexes, and segment-management work. Larger segments reduce the number of files but can keep older records alongside newer ones for longer before a whole segment becomes eligible for removal.

Consider a partition replica retaining approximately 1 GiB of data. Ignoring the partially filled active segment and assuming size-based rolling dominates, 64 MiB segments give roughly 16 data files. With 256 MiB segments, the same data occupies roughly four.

Across 600 partition replicas, those estimates become 9,600 versus 2,400 data files, plus the supporting files. The difference does not mean smaller segments store four times as much event data. It means the broker and filesystem manage the same history in more files.

Traffic matters just as much as the configured size. At a steady stored-data rate of 2 MiB per second, a 256 MiB segment takes roughly 128 seconds to fill. At 2 KiB per second, it takes roughly 36 hours. A time-based condition may roll the quiet partition much sooner, leaving smaller files despite the same size setting.

These are estimates based on bytes Kafka actually appends after compression, excluding index overhead. Bursty traffic, batch sizes, and other roll conditions change the result. A topic can contain both busy and quiet partitions, so a topic-wide throughput average can hide very different segment behavior.

Choose settings by considering the cleanup precision the application needs and the number of local replicas each broker holds. Then observe actual file counts, roll frequency, retained bytes, and disk activity. A smaller value is useful only if its cleanup benefit justifies the extra file-management work.

Segment size is also not a direct consumer-latency setting. Readers can consume committed records while a segment is active; they do not wait for it to fill or roll. Reducing segment.bytes to make fresh events visible sooner addresses the wrong mechanism.

After a restart, Kafka discovers segment files, checks their associated state, and recovers data where needed. Checkpoints and producer-state snapshots help limit reconstruction work, while valid batches allow Kafka to rebuild supporting indexes.

Suppose a broker rolls at 250, then fails while writing the first batch into the new segment. If the failure leaves an incomplete batch on storage, local recovery can discard that invalid tail. The older segment ending at 249 remains a separate portion of the log that Kafka can check and retain.

This organization makes the files manageable, but a roll is not a durability boundary. It does not prove the old segment's records survived on enough replicas, and it does not make every byte in the new segment committed. Recovery still depends on valid local data and synchronization with the current partition leader.

Smaller segments also do not guarantee a faster restart. More files add loading and bookkeeping work, while the amount of log data requiring validation depends on the shutdown and persisted recovery state. Evaluate recovery using the broker's actual workload and failure conditions rather than segment size alone.

Kafka divides each partition log into segments, with one active segment receiving normal appends. Rolling creates a new append destination while offsets continue across the boundary and older records remain readable.

Size, time, and internal limits can trigger a roll. Segment boundaries determine how finely Kafka can manage cleanup, creating a trade-off between retained history and file-management overhead. They do not change partition ordering, consumer assignments, or the conditions for replication commitment.