A Kafka consumer asks for a topic, a partition, and an offset. The broker must turn that logical address into bytes it can read from storage. Understanding how Kafka organizes those bytes makes broker directories easier to inspect and helps explain what happens when a broker writes, reads, or recovers records.

We'll follow a batch of order events into a partition's files, then trace how Kafka finds those records again. Our example uses a KRaft-based Kafka cluster, the modern record batch format, and ordinary local filesystem storage. The cluster does not use tiered storage, and the example records are non-transactional and cleanup has not removed them.

A broker stores topic data under the directories log.dirs lists. For example, a broker might use this storage setting:

This is an illustrative setting, not a complete broker configuration. The path belongs to the machine or container running the broker. In a container deployment, the corresponding host path depends on how you mount the volume.

Inside a data directory, Kafka normally gives each local partition replica its own directory, whose name combines the topic and partition number. A replica of partition 0 of orders.placed therefore lives under orders.placed-0.

Suppose Broker 1 stores replicas of partitions 0 and 1. The following layout shows two segment data files for partition 0; this diagram leaves out supporting files.

The directory contains Broker 1's copy of the partition. With a replication factor of three, two other brokers store their own copies in their own data directories. A broker stores the replicas Kafka assigns to it, which may include both leaders and followers.

When log.dirs lists several locations, a partition replica normally occupies one of them. Kafka does not stripe each incoming record across every configured directory. Adding storage paths also does not increase the topic's replication factor.

These data directories serve a different purpose from the broker's diagnostic logs, such as server.log. Diagnostic logs describe what the Kafka process is doing. Partition files contain the records applications publish.

KRaft also maintains a cluster metadata log. The metadata.log.dir setting controls its location, which falls back to the first configured log directory when unset. Cluster metadata and application events have separate logs even when their directories share a storage device.

A partition's log can grow for days or months. Kafka divides it into log segments, manageable portions of the log that Kafka stores in separate files. Each segment has a data file and supporting indexes. The active segment receives new appends; Kafka creates another segment when it needs to roll the log forward.

For our example, suppose partition 0 has an older segment covering offsets 0 through 99 and an active segment that began at offset 100. We use these boundaries to make the example readable. Kafka does not normally roll a segment every 100 records.

The filenames use the segment's base offset, which Kafka writes as a 20-digit decimal number with leading zeros. A base offset identifies where that segment begins in the logical log. It does not report a byte position or the number of records in the file.

Here are the main files associated with the example's active segment:

The shared prefix associates the files with the same segment. The older segment has its own files with the prefix 00000000000000000000.

The .log file holds the record data. An index contains lookup information, not another full copy of the events. Kafka can rebuild offset and timestamp indexes by examining valid log data; an index cannot reconstruct a missing event payload.

A real directory can contain additional files. For example, .txnindex files track aborted transaction information, producer-state snapshots help restore producer tracking, and leader-epoch-checkpoint records leadership history that replicas use to synchronize. They support the log's operation and recovery. You should not expect every partition directory to contain only the three file types in the table.

Consumer progress is separate again. Kafka-managed group offset commits go to the internal __consumer_offsets topic. Processing an order does not add a per-consumer completion flag to its .log entry.

Even when an application publishes JSON, a Kafka .log file is a binary file. The JSON bytes sit inside Kafka's record encoding, alongside lengths, timestamps, keys, and other metadata. Opening the file in a text editor does not give you a reliable list of events.

Kafka stores records in record batches. A batch contains one or more records for one partition, with metadata those records share. A segment can contain many batches, and a single batch does not span partitions.

Suppose a producer groups three OrderPlaced events for partition 0 into one batch. The leader assigns offsets 120, 121, and 122. The leader appends the batch to the active segment whose base offset is 100.

The records have these application fields. The table abbreviates the values; each record also carries a header named event-type whose value is the UTF-8 encoding of OrderPlaced.

The batch's base offset is 120. Within it, the records encode offset deltas of 0, 1, and 2. Adding each delta to the batch's base offset gives the record's absolute offset.

This diagram shows the relationship between the segment, batch, and individual records. It is a conceptual view, not a byte-for-byte layout.

There are three different positions here: the segment begins at 100, the batch begins at 120, and a record within that batch has its own offset. None of these numbers tells you how many bytes to skip from the start of the file.

The batch header includes its length, format version, offset and timestamp information, compression attributes, and a CRC-32C checksum for detecting corruption. It also carries fields that Kafka uses to track producers and transactions. Individual records contain their key and value bytes, headers, and offset and timestamp deltas.

Application headers such as event-type are distinct from the batch's structural header. Kafka understands the batch structure but does not interpret the business meaning of the order's fields.

When you enable compression, Kafka compresses the records together within a batch. The batch metadata describes the encoding so readers can interpret it. Compression does not combine an entire partition or segment into one compressed object, and each record keeps its logical offset.

The producer serializes keys and values before sending them. The broker receives encoded batches, validates their structure and applicable limits, and appends accepted data to the partition log. On the leader, this processing also assigns the partition offsets.

Kafka's network and storage formats both use record batches. That lets the broker handle data in batches without turning every event into an application object and then serializing it again. The broker can still inspect or modify batch metadata, and compression settings can require recompression, so the broker does not always preserve the incoming bytes exactly.

Assume the active segment already contains records through 119 and has room for our new batch. The append adds the batch containing 120 through 122 to 00000000000000000100.log. The local log end offset becomes 123, the next position after the appended records. Kafka maintains supporting index entries as needed; there is not necessarily an index entry for every record.

At the operating system level, writing the file usually updates the page cache, memory the operating system uses to cache file contents. The operating system eventually writes modified pages to the storage device. The diagram separates the filesystem append from that later writeback.

The page cache holds cached contents of the file. It is part of the filesystem write path, rather than a separate application queue that Kafka must later drain into a file.

A completed append does not normally imply that Kafka forced those bytes onto physical storage with fsync. Producer acknowledgment settings govern what the broker must confirm about the write and replication; acks=all is not a request to flush every replica's disk for every batch.

Followers fetch the partition's batches and append them to their own local logs, preserving the record offsets. Each replica has its own filesystem writes and flush timing. Replication helps protect against individual broker failures, while losing unflushed data across all usable copies is a different failure scenario.

Suppose a consumer requests partition 0 beginning at offset 121. It sends a logical offset, not a filename or a disk address. The broker handles the translation.

In our layout, offset 121 belongs in the segment beginning at 100. Kafka uses the offset index to find a nearby physical position in that segment's .log file, then scans forward through batches to locate the relevant data. The index is sparse, meaning it records selected positions rather than an entry for every offset.

Because 121 sits inside the batch beginning at 120, the broker can return the containing batch. The consumer client decodes it, decompresses its records if necessary, and delivers records from the requested position. The application does not need to understand segment filenames or batch boundaries.

Record sizes vary, and compression changes the relationship between logical records and stored bytes. You cannot calculate the file position of 121 by multiplying it by a fixed record size. Even the difference 121 - 100 describes an offset distance, not a byte distance.

Reading the file does not always cause a physical device read. If the requested bytes are in the page cache, the operating system can serve them from memory. If they are absent, storage must supply them. A consumer following recent writes and a consumer replaying old history can therefore place different demands on the same broker.

File contents also do not determine visibility by themselves. A batch may be physically present in the leader's log before replication makes it available to consumers. For our non-transactional batch, a high watermark of 123 allows reads through 122. While the high watermark remains at 120, the batch at 120 through 122 is outside the readable committed range.

Reading leaves the stored batch in place. Other consumers can read the same bytes, and cleanup policies determine when Kafka removes them.

After a broker restarts, it needs to establish which stored batches are valid and rebuild the in-memory state needed to serve its replicas. Kafka uses persisted checkpoints and snapshots to avoid reconstructing everything from scratch, and scans log data where recovery requires it.

Batch lengths describe the boundaries of encoded data, while checksums help detect damaged contents. If a crash leaves an incomplete batch at the end of a file, recovery can discard the invalid tail instead of interpreting those bytes as complete records. Kafka can also rebuild supporting indexes from valid batches.

For example, suppose the file contained complete batches through 119 before the broker began appending our batch. If a machine failure leaves only part of the batch for 120 through 122 on storage, a few readable fragments of JSON do not make those events valid Kafka records. Recovery must establish a valid batch boundary.

Local file recovery and replica synchronization answer different questions. A locally valid batch may still belong to an uncommitted history that conflicts with the current leader. A recovering follower must reconcile its log with that leader and fetch missing data. Conversely, data missing locally may be recoverable from another replica.

Checksums detect corruption; they cannot recreate lost payloads. Whether the cluster can restore the records depends on the surviving replicas and the state they contain. Manually deleting or editing a segment file bypasses the relationships among data, indexes, and replica state, so do not treat file inspection as a repair procedure.

Kafka includes kafka-dump-log.sh for inspecting its binary log files. This is useful when you need to check batch offsets or understand a segment's contents without guessing from the filename.

For a local inspection, use an Apache Kafka binary distribution compatible with the broker's file format and a Java runtime that distribution supports. Run the command from the distribution directory against a consistent copy of a segment, such as a copy you take after shutting down a local learning broker. Replace the path with the actual copied file:

The command reads a local file; it does not connect to a running broker. It reports batch information that lets you compare the batch's starting and ending offsets with the segment base offset in the filename. Exact output fields depend on the Kafka version.

In our example, the filename’s numeric prefix represents base offset 100, while one of the batches inside begins at 120 and ends at 122. Those values are consistent: the file holds multiple batches after the segment's starting position. Your own topic will have different offsets and may have only one segment if producers have written little data.

Inspecting batch structure is also different from decoding application data. JSON that uses UTF-8 encoding, Avro, and Protobuf produce different value bytes. Understanding a stored value requires the application's serialization format and, where applicable, its schema. The binary log format supplies the record boundaries; the application format supplies the meaning.

Each broker stores its assigned partition replicas in data directories. A partition log consists of segment data files and supporting indexes, and each data file contains binary record batches with their own metadata and individual records.

Writes append batches through the filesystem, usually using the operating system's page cache. Reads translate logical offsets into file positions, while recovery checks stored data and reconciles replicas. Segment offsets, record offsets, byte positions, and replication commitment describe different parts of that process; keeping them separate makes Kafka's disk layout much easier to understand.