In this lab, you will fill a one-partition topic with a few thousand order events, open the partition directory, and read the segment, index, and timestamp files with kafka-dump-log.sh. You will then watch retention delete segments, watch compaction rewrite one, look for page-cache effects, and corrupt an index file to see the broker rebuild it.

The fixed contract is a generated input file of exactly 8,000 keyed records with a known padding size, which you publish to orders.placed with segment.bytes=1048576. Use the known input size and observed batch layout to estimate segment counts and predict retention, then check the files on disk. Phase 4 uses a separate keyed input to predict compaction results.

Use environment E1, the single broker container kafka-local running apache/kafka:4.3.1. Every Kafka tool runs inside the container through docker exec, and the data lives under /tmp/kraft-combined-logs in the container's filesystem. On the host you need Docker, a POSIX shell, and the shell's time keyword. You do not need to install any additional tools on the host.

Use a fresh lab container and fresh topic names for this experiment. If an earlier lab left kafka-local, finish its cleanup before running:

Create the folders and change into the lab directory. Run all remaining host commands from this directory, and use it in any additional terminal. Generate the input file. Each line is a key, a pipe, and a JSON value with 200 padding characters so that 8,000 records occupy roughly 2.5 MiB:

Confirm the file has 8,000 lines with wc -l and record its byte size. You will compare that size with the partition directory later.

Create the topic with one partition and a 1 MiB segment limit. Set the index interval explicitly to its default value so you can change it in an extension:

Produce the whole file through the console producer:

List the directory and its size:

Save the listing to observations/phase1-listing.txt and annotate every file: its base offset, whether it is a .log, .index, or .timeindex, and whether its segment is active or inactive. Note any other files, such as producer-state snapshots, leader-epoch-checkpoint, or partition.metadata.

What you should see: three .log files, two near 1 MiB and one smaller active segment, with 20-digit names whose numbers rise. The first is 00000000000000000000.log. The .index and .timeindex of the active segment show a preallocated size far larger than the inactive ones, because Kafka trims active indexes only when the segment rolls. The du total is close to the input file size plus index overhead, because the batches are uncompressed.

Dump the first segment with record payloads. Replace the filename with your own if it differs:

Each batch line reports baseOffset, lastOffset, count, position, CreateTime, size, compresscodec, and isvalid. Each record line under it reports offset, keySize, valueSize, key, and payload. Copy the first two batches into observations/dump-log-excerpts.md and verify that adding the batch's base offset to each record's position within the batch gives the printed record offset.

Now dump the two indexes of the same segment:

Count the offset-index entries by piping the full index dump through wc -l and subtracting the header line. Divide the .log file size by 4,096 and compare. Then pick one index entry, find the batch in the .log dump whose position matches it, and record which offset in that batch the entry names.

Finally, add 1,000 compressed records and dump the newest segment:

If the tool rejects the codec argument, check --help and write it as --compression-codec=zstd.

What you should see: batches of several records each with compresscodec: none in the first segment and compresscodec: zstd in the newest one, with a compressed batch's size well below an uncompressed batch of the same count. The offset index holds roughly one entry per 4 KiB of log data, so a 1 MiB segment has a few hundred entries for thousands of records. Each entry's offset is the last offset of a batch and its position is that batch's start. The timestamp index has the same number of entries or fewer, and its timestamps never decrease.

The diagram shows the three things that can happen to an inactive segment in the rest of this lab. Look for which file names survive each path.

Before changing anything, write a prediction in observations/retention-timeline.md. Using your ls -l sizes, apply the rule from the retention chapter: Kafka removes the oldest segment while the total size minus that segment's size still meets retention.bytes. Set a limit of 1.5 MiB and name the segments you expect to disappear.

The broker checks retention every log.retention.check.interval.ms, which defaults to 300,000 ms, so the next retention check may take about five minutes; physical deletion follows later and other conditions can delay it. Poll the directory every 30 seconds and note the time of each change:

Stop once you confirm the predicted segments are gone. Record any intermediate rename you capture, and use broker logs for transitions the polling misses. Then read the broker's own account:

Now add time-based retention while leaving the size limit enabled, and wait again:

Run the polling loop until the directory settles, then check both ends of the partition:

What you should see: after the size limit, the oldest .log and its two indexes first appear with a .deleted suffix, then vanish about a minute later, while the log lines name the deleted segment's baseOffset, cite a retention size breach, and report the incremented log start offset. After the time limit, every segment holding data expires, yet the directory still contains one empty .log whose base offset equals the log end offset. The earliest and latest offsets printed by kafka-get-offsets.sh are equal. Kafka rolled a fresh active segment before deleting the old one, because a partition always needs an append destination.

Restore the defaults and reload the data for the remaining phases:

Re-run the Phase 1 producer command with the same input file. List the directory again and record the new base offsets. They continue from the previous log end offset rather than restarting at zero, and raising retention did nothing to bring the deleted files back.

Create a compacted status topic with aggressive cleaning and a short tombstone retention:

Produce seven records, including a tombstone for ord-1044. The null.marker reader property turns the literal text NULL into a null Kafka value:

Read everything back and save the output as the "before" section of observations/compaction-before-after.md:

The cleaner never compacts the active segment, so the seven records must leave it. Wait at least 15 seconds, then append one more record, which makes Kafka evaluate the time-based roll:

Wait about a minute, then re-run the consumer and inspect the segment. Cleaning runs in the background; if the older values remain, repeat these observations until you see a cleaning pass. Save the actual surviving offsets rather than assuming that a minute is a deadline:

After observing the first cleaning pass, wait a further 60 seconds, then append ord-1045|{"status":"PAID"} with the same one-line producer. Read again and record the raw records for the "after" section. Repeat the read and segment inspection until a later cleaner pass removes the expired tombstone; the delay also depends on cleaner eligibility and scheduling. If it remains, report that state and inspect the cleaner logs instead of claiming the wait guaranteed deletion.

What to verify: the first read prints offsets 0 through 6, including the null value for ord-1044 at offset 6. After cleaning, superseded values disappear and surviving records keep their original offsets. The segment filename can still start at 0 even when its first surviving record has a higher offset. A retained tombstone batch can show deleteHorizonMs; a later eligible pass can remove it after that horizon. Record whether each snapshot still contains it.

After the last append, reconstruct current state by applying records in offset order: ord-1042 is SHIPPED, ord-1043 is CANCELLED, ord-1045 is PAID, and the tombstone removes ord-1044. The raw read may still contain both PLACED and PAID for ord-1045: the new PAID record is in the active segment, which the cleaner does not compact. Do not confuse a correct reconstructed state with a log that already contains only one record per key.

The reloaded orders.placed holds 8,000 records. Record the container's memory line before reading:

Time a full read three times in a row, discarding the output:

Run docker stats --no-stream kafka-local again and write all numbers into observations/cache-timing.md.

What you should see: three similar timings dominated by JVM startup, and a memory figure that barely moves. Treat this as illustrative. Recent writes may still be cached, but these timings do not prove cache residency or exclude misses. Use storage-read and cache-pressure measurements to distinguish warm and cold reads. Explain in the report which of the observations from the page-cache chapter's evidence table you could make here and which would need a broker whose retained data exceeds its memory.

From the current listing of orders.placed-0, pick the lowest base offset. It belongs to an inactive segment. Set it as a variable using your own value:

Stop the broker cleanly, then overwrite that segment's offset index with 17 bytes of text. Any length that is not a multiple of 8 fails the index sanity check. The copied file must allow the broker user to rewrite it because docker cp creates it as root while the broker runs as an unprivileged user that needs to rewrite it:

Wait for startup, then collect the evidence:

Save the log lines and listing to observations/index-recovery.txt, then re-run the Phase 5 read once to confirm all 8,000 records are still readable.

What you should see: a WARN line naming the segment's .log file, describing a corrupted index, and stating that the broker is recovering the segment and rebuilding index files. The .index file's size is a multiple of 8 again and its modification time is after the restart, the sanity check passes, and the consumer still reads 8,000 records. The broker did not need a replica because Kafka derives an index from the batches in the .log. Damaging the .log itself would have been a different outcome.

Verify with docker ps -a --filter name=kafka-local, which should print only the header line. Keep the observations and report.

The files under orders.placed-0 are the commit log the module describes. Each segment's filename carries its base offset, its .log holds binary record batches whose offsets you can add up from the batch base, and its two indexes hold sparse entries spaced by index.interval.bytes that point at batch boundaries rather than individual records. Retention removed whole segments, renamed them before deleting them, advanced the log start offset, and still left one active segment behind. Compaction wrote replacement segment files, kept the latest value per key with its original offset, and retained the tombstone until the retention conditions allowed a later cleaner pass to remove it.

The last two phases showed the limits of local evidence. This timing-only experiment cannot establish whether a read caused cache misses, so timing alone says little about the page cache, and the chapter's evidence table tells you what else to measure. A corrupted index cost the broker a rebuild at startup and nothing more, because Kafka derives indexes from valid batches; the .log data is the only copy of the records on a single broker, which is why replication, not file repair, is Kafka's answer to losing it.