When you inspect an order event in Kafka, the JSON you see is usually only its value. The record also has a key, a timestamp, and possibly headers. Once Kafka stores it, it has a position in a particular topic partition.

These parts answer different questions: Which order is this about? What happened? When did it happen? Where can we find the stored record? In this chapter, we’ll take apart one order record and follow how its contents and metadata move from producer to consumer.

A record is the unit of data an application writes to and reads from Kafka. People also call it a message. An event is the business fact that a record may describe, such as a customer placing an order. Records can carry other kinds of data too, including commands and snapshots of current state.

Suppose the order service publishes an OrderPlaced event for order ord-1042 to orders.placed. Its key is ord-1042, and its value is this JSON document:

The amount is in paise, so 129900 means INR 1,299.00. These field names and their meanings belong to the application’s event contract. Kafka does not define them.

Assume the producer explicitly supplies the event time as the Kafka timestamp and attaches two headers. The producer routes the record to partition 0, where Kafka assigns offset 42. This is an illustrative position, not an offset predicted from the key.

Here is the record as a consumer might inspect it. The example shows timestamp and header values as readable text:

The diagram separates the application data from the metadata that describes its location and timestamp. It shows the logical view clients expose, not Kafka’s byte layout on disk.

The JSON is one part of the record. Adding a field named key, timestamp, or partition inside that JSON would not set the corresponding Kafka field. The producer must supply Kafka fields through its client API or tool.

The key is an optional value Kafka stores separately from the record’s main value. It often identifies the entity the data concerns: an order ID, account ID, or device ID.

In our example, the key and the JSON orderId both contain ord-1042. That repetition is intentional. Kafka clients can use the key without parsing the JSON, while the event body remains understandable on its own. The producer is responsible for keeping them consistent.

Keys commonly influence partition selection. With the standard Java producer’s key-based routing, the same serialized key maps to the same partition as long as the partition count and routing behavior remain unchanged. An explicitly selected partition or a different partitioning configuration can change that behavior. A key is therefore useful for keeping related records together, but it is not an unconditional ordering guarantee.

A key is also not a database uniqueness constraint. Publishing another record with key ord-1042 normally appends another record; it does not overwrite the existing one. Multiple events about the same order can legitimately share that key.

For a topic using log compaction, Kafka uses keys to determine which older records it can remove as newer state becomes available. That cleanup happens later. Even there, publishing the same key is an append, not an immediate update in place.

Records can have a null key when the topic and application permit it. For example, a stream of independent diagnostic events may not need an entity key. A compacted topic requires non-null keys because cleanup needs to identify which records describe the same entity.

The value carries the record’s main content. In our example it describes an accepted order, but Kafka does not require JSON. Applications might use text, Avro, Protobuf, or another agreed representation.

Before sending a record, the producer converts its key and value into bytes. This is serialization. The consumer converts those bytes into usable values through deserialization. The key and value have separate serializers and deserializers, so they do not need to use the same format.

For this example, assume the key is a UTF-8 string and the producer encodes the JSON value as UTF-8. UTF-8 is a standard way to represent text as bytes. A consumer could decode the value into a string and then parse that string as JSON, or use a deserializer that performs both steps.

The diagram follows these two fields through the system. The diagram omits headers and location metadata to keep the conversion clear.

Kafka checks its record format and applicable broker limits, but it does not interpret totalMinorUnits or validate our order schema. A successful write can therefore contain data that a consumer cannot understand.

For example, valid UTF-8 text is not necessarily valid JSON. Valid JSON can still violate the contract by putting a string where the consumer expects a number. And a correctly shaped event can carry the wrong business meaning, such as an amount expressed in rupees when the contract requires paise.

These are different failures. Deserialization may fail before business processing starts, while contract validation may reject an already decoded value. Repeatedly reading the same stored bytes will not repair them; the application needs a deliberate way to handle the failure.

The producer and consumer do not have to use the same programming language. They do have to agree on the bytes and their meaning.

Headers carry additional metadata alongside the key and value. Each header has a string name and a value that contains bytes or is null. A record may have no headers.

Our example uses these application-defined conventions, and encodes header values as UTF-8:

The content type tells cooperating applications the value’s format. The correlation ID helps connect this event with other activity from the same checkout request, such as logs and downstream operations.

Kafka does not automatically select a JSON deserializer because a header says application/json. A client library or application must explicitly implement that convention. Likewise, a correlation header is useful only if the participating applications preserve and use it.

Headers are an ordered collection, and names can repeat. If two headers have the name correlation-id, treating them as a simple map may discard information. An application contract should say whether it allows repeated names and which value to use when they occur. Kafka’s Java header API supports reading all matching headers or the last matching header.

Choose where each metadata field belongs. In this example, orderId, amount, currency, and occurredAt are in the value because they describe the business event. The correlation ID is in a header because it helps trace processing. Other arrangements can work, but consumers need one clear contract rather than conflicting copies of the same information.

Headers travel with the record. Moving sensitive data from the value into a header does not hide it from an application that can read that record.

A modern Kafka record has a timestamp that counts milliseconds since the Unix epoch, January 1, 1970 at 00:00:00 UTC. Tools may display it as a date and time. The topic setting message.timestamp.type determines whether Kafka uses CreateTime or LogAppendTime.

With CreateTime, Kafka uses the timestamp the producer supplies. In the Java producer, if the application does not supply one, the producer assigns its current time. The name does not guarantee that this is when the business event happened.

With LogAppendTime, the broker replaces the producer timestamp with the time it appends the record to its log. This describes arrival at Kafka, which may be later than the event itself.

Consider this timeline for our order. All times are on September 7, 2026, in UTC:

The event time, append time, and processing time describe different moments. For the record we inspected, the application explicitly set the Kafka timestamp to 09:30:00, and the topic uses CreateTime. That is why its timestamp matches the JSON occurredAt field despite the two-second delay before sending.

If the application instead let the Java producer supply its current time at 09:30:02, CreateTime would reflect that time. If the topic used LogAppendTime, the stored timestamp would be 09:30:03. In all three cases, the JSON occurredAt would remain 09:30:00; Kafka does not rewrite fields inside the value.

Kafka does not automatically add the consumer’s processing time to the stored record. A reporting application measures that separately. Replaying the record tomorrow changes its processing time, not its original stored timestamp.

Timestamps do not determine log order. Kafka can append a delayed event after an event with a later timestamp. Offsets describe positions in the partition; timestamps describe time according to the configured policy and the clocks supplying it.

The producer prepares a record before Kafka knows where it will sit in the log. It supplies the destination topic, key, value, headers, and optionally a partition and timestamp. If the application supplies no partition, the producer selects one using its routing behavior.

When the broker appends the record, it assigns the offset. The application does not choose offset 42 as a producer input. A consumer receives the stored data together with the topic, partition, offset, and timestamp information needed to interpret its source.

For our record, three identifiers are especially useful:

A later event about the same order can keep ord-1042 as its key while receiving a different event ID. A second publication of the same business event can keep evt-7001 but receive a different Kafka offset. Kafka does not inspect the JSON event ID to reject that second publication.

Producer retry protections can prevent duplicates caused by certain retries. They do not turn an application’s eventId field into a uniqueness constraint. A stable event ID helps applications recognize repeated events only when their processing logic actually checks it.

Replay is different from republishing. Reading the retained record at orders.placed, partition 0, offset 42 again returns the same record at the same position. Republishing its contents creates a new record with a newly assigned position, even if the key and value are identical.

The record itself has no shared “processed” flag. If reporting finishes offset 42 and all earlier records, it may commit 43 as its next restart position. That checkpoint belongs to the consumer group; it does not modify the record or prove that another application has completed its work.

A Kafka value can be null. This is different from a non-null value containing the text null, an empty string, or a JSON object with a null field.

For example, this is an ordinary JSON value with a missing cancellation reason:

The entire document still becomes a non-null sequence of bytes. It is not a Kafka null value.

In a compacted topic, a record with a non-null key and a null value is a tombstone: a deletion marker for that key. To illustrate, consider a separate topic, orders.current-state, that uses compaction:

This marker allows compaction to remove earlier state for ord-1042 in that partition. Removal happens through cleanup, not immediately when the producer sends it. Kafka can also remove the tombstone itself once its retention rules allow removal.

A consumer maintaining a current-state table would normally interpret the marker as an instruction to remove the corresponding row. Kafka does not perform that database deletion for the consumer. The application must distinguish a tombstone from malformed data before attempting to parse the value as JSON.

A null value on a topic using only time- or size-based deletion does not trigger compaction. That topic’s contract must define what the null value means. Our orders.placed event contract expects an event body, so a null value would need explicit handling as unexpected input.

The record fields applications inspect are a logical view. Kafka’s storage format groups one or more records for a partition into a record batch. Even a batch containing one record has batch metadata.

The batch holds information such as a base offset, timestamp information, and a checksum for detecting corruption. Individual records encode fields including their keys, values, headers, and offset and timestamp differences relative to the batch. Topic and partition identity come from the surrounding request or storage context; Kafka does not repeat the topic name inside every encoded record.

When you enable compression, Kafka compresses record data at the batch level. The consumer client decodes the batch and exposes individual records with their own offsets, keys, values, and headers. Batching does not merge several order events into one business event.

The word “header” can refer either to this internal batch metadata or to the application headers a producer attaches to a record. They serve different purposes: a batch checksum helps Kafka validate stored data, while correlation-id helps our application trace an event.

Applications normally work through client APIs rather than parsing this binary format themselves. Understanding the distinction explains how Kafka can store and transfer data efficiently while still letting consumers inspect each record separately.

A Kafka record carries a key, value, timestamp, and optional headers. The producer supplies its contents and destination, serialization turns the key and value into bytes, and Kafka assigns a position in a partition. Consumers see both the data and its location metadata.

Keep each field’s role clear. The key identifies related data, an application event ID identifies a business event, and an offset identifies a stored position. Timestamps need a defined meaning, headers need an agreed format, and null values need deliberate handling. These distinctions make records easier to publish, inspect, and process correctly.