An order service works with strings, numbers, and objects. Kafka stores record keys and values as bytes. A serializer connects those two representations, deciding exactly what the application puts into a record.

That decision becomes part of the agreement between producers and consumers. A producer can successfully publish bytes that a consumer cannot understand, and changing how the serializer encodes a key can change where its records go.

In this chapter, we’ll examine that agreement, compare Kafka’s built-in serializers, and implement a small JSON serializer for an order event. The examples use Java 21 and the Apache Kafka Java client 4.3.1. We’ll test the encoding directly, without needing a running broker.

A serializer converts an application value into bytes. A deserializer interprets bytes and reconstructs a value that the receiving application can use.

Suppose an order service creates an OrderPlaced object and publishes it to orders.placed, using ord-1042 as the key. The key serializer encodes the order ID. The value serializer encodes the event’s fields. Kafka receives those byte sequences inside a record, along with metadata such as its timestamp and headers.

The diagram follows the value through this process:

Kafka understands the record’s structure, but an ordinary broker does not need to know that the value describes an order. It does not automatically check that currency is valid or that the consumer expects JSON. Successful storage and successful interpretation are separate outcomes.

The producer and consumer do not need to use the same programming language or Java class. They need to agree on the wire format: the representation of the data as bytes. A Python consumer can read a Java producer’s JSON if both applications agree on the encoding, field names, types, and meanings.

A serializer handles a key or value, not the entire Kafka record. The client handles record framing and metadata. Each header has its own name and a value containing bytes; the value serializer does not automatically encode them as part of the event.

Kafka’s Java client provides serializers for common types. These are useful when the application already has the representation it wants to publish.

Big-endian byte order means “most significant byte first.” It is part of the format a reader must understand.

The input type matters. StringSerializer does not call toString() on an arbitrary object. It expects a string. Likewise, LongSerializer writes a binary integer; it does not produce the text representation of a number.

Consider the value forty-two. The string "42" becomes the two bytes 34 32 in hexadecimal. The Java long 42L becomes eight bytes: 00 00 00 00 00 00 00 2a. Hexadecimal is simply a readable way to display the bytes; those space-separated characters are not what Kafka stores.

Both representations can describe the same number to an application, but they are different formats. A LongDeserializer rejects the two-byte string representation because it expects eight bytes. A mismatched decoder can also produce an incorrect value without an obvious error, so “it did not throw” is not a sufficient compatibility check.

ByteArraySerializer is useful when another library has already produced the intended bytes. It does not tell consumers what those bytes mean. That agreement still belongs to the applications.

The producer configures key.serializer and value.serializer separately. This lets an order event use a simple string key and a structured value.

For example, a KafkaProducer<String, OrderPlaced> works with a string key and an OrderPlaced value. Its configured serializers must accept those types. Java’s generic declaration helps callers use the producer correctly, but it does not validate arbitrary serializer class names in configuration.

The key deserves particular care. With the Java client’s normal keyed routing, when the application supplies no partition and has not configured the producer to ignore keys, partition selection uses the serialized key bytes. Changing a numeric identifier from UTF-8 text to a binary long changes the input to that selection, even if the business identifier still means the same thing.

Such a change can move subsequent events to another partition. It does not always do so, because different byte sequences can select the same partition. The important point is that keeping the logical ID unchanged is not enough to preserve routing.

Key bytes also determine key identity for log compaction. On a compacted topic, two encodings of the same business ID are different keys. A change to key serialization therefore needs coordination beyond updating a Java type.

Values have their own compatibility risks. Changing a field from an integer amount in paise to a string amount in rupees can break consumers even if both payloads remain valid JSON. Serialization preserves a representation; it does not preserve meaning automatically when the application changes that representation.

If an application already has a JSON string, StringSerializer can encode it. When the application works with an event object, a custom serializer can put the object-to-JSON conversion in one place.

We’ll use Jackson 2.20.0 to generate JSON. Jackson is a separate library, not a built-in Kafka serializer. Pinning it here makes the example reproducible. The example uses its 2.x API and package names.

Create a small Maven project:

Save this as pom.xml. The first build needs internet access to download the dependencies and plugins.

All three Java files below belong in src/main/java/com/example/kafka.

The Event Type

Save this event definition as OrderPlaced.java:

We use an ISO-8601 timestamp string to keep the example focused on serialization rather than date-library configuration. The amount is an integer in the currency’s minor units. For INR, 129900 means 129,900 paise, or INR 1,299.00.

This simple record does not validate business rules. An application should validate required identifiers, allowed event types, amounts, and timestamps before treating an event as ready to publish. Being encodable is not the same as being a valid order event.

The Serializer Implementation

Save this as OrderPlacedSerializer.java:

Serializer<OrderPlaced> declares the accepted input type. Its serialize() method returns the bytes Kafka should use for that field. The topic argument is available if an implementation needs topic-specific behavior; this serializer deliberately uses one format for every call.

Jackson’s writeValueAsBytes() produces UTF-8 JSON directly. It handles escaping quotes, backslashes, and other characters, avoiding the fragile string concatenation that often appears in early producer implementations.

The serializer creates the mapper once and does not reconfigure it after use begins. This matters when application threads share a producer and invoke serialization concurrently. A serializer should avoid shared mutable buffers or other per-record state that one call could overwrite while another call is using it.

If encoding fails, the implementation preserves the original exception as the cause of a SerializationException. It does not substitute an empty value or silently skip the event.

Serialization is a local operation, so we can inspect its result without starting a broker or publishing records. This separates a format problem from connection, permission, or delivery problems.

Save the following as SerializerDemo.java:

Run it from the directory containing pom.xml:

Alongside Maven’s output, expect:

The check confirms that the bytes decode back into the original event and that null input follows the serializer’s chosen behavior. The JSON’s field order is not part of the application contract; consumers should read fields by name.

A round trip through the same library is a useful first check, but it is not proof that every consumer is compatible. For a shared event format, also check representative bytes with the actual consumer implementation. Include escaped text, non-ASCII characters, boundary-sized numbers, and retained records from older producers. These cases expose assumptions that a single ordinary event can miss.

To publish OrderPlaced objects, the producer needs a string key serializer and our value serializer. These are the relevant entries in an existing producer configuration:

The custom class must be available on the application’s classpath. When Kafka creates a serializer from configuration, it needs a no-argument constructor; our public class has an implicit public one. The producer’s types should be KafkaProducer<String, OrderPlaced>, and the submitted record’s types should be ProducerRecord<String, OrderPlaced>.

These two entries are only the serialization configuration. A running producer still needs broker connectivity, delivery settings, result handling, and lifecycle management.

Kafka calls configure(configs, isKey) on serializers it creates from configuration. The isKey flag identifies which field the instance handles. Our serializer needs no additional settings, so the interface’s default method is sufficient. If you supply serializer instances directly to the producer constructor, configure them yourself when necessary; the producer does not call their configure() method in that case.

The interface also provides a header-aware serialize(topic, headers, data) method. Its default implementation delegates to the simpler method this example uses. A specialized implementation can examine headers, but setting a header such as content-type=application/json does not make Kafka choose a JSON serializer or decoder automatically.

The producer closes its serializers when it closes. Implementations that own resources can release them in close(), which should tolerate repeated calls. Our implementation has no resource it must explicitly close, so the default method is enough.

Consumers must still know how to read the resulting value. StringDeserializer turns these JSON bytes into a Java string, after which an application can parse it. A JSON-aware deserializer can instead return an event object directly. The topic name does not automatically select either approach.

There is also an important double-encoding trap. Passing an already-generated JSON document to a general JSON encoder as a string can produce a JSON string containing a document, with extra quotes and escaped characters. If the intended value is a JSON object, encode the object once, or send already-produced JSON bytes through ByteArraySerializer.

The custom serializer returns Java null for a null event. That represents a null Kafka value, which is different from an empty byte array or a textual JSON value.

On a compacted topic, a record with a non-null key and null value acts as a tombstone, marking that key for deletion through compaction. It does not immediately erase every earlier record. On a topic that only uses deletion-based retention, a null value does not trigger key-based cleanup.

This is why a serializer must not catch an encoding error and return null as a convenient fallback. On a compacted topic, that fallback could turn a failed update into a deletion marker.

Our serializer preserves null input so that the distinction remains explicit. That does not mean an orders.placed application should accept a missing event. The topic’s application contract must state whether it permits null. Validate that rule before sending. A null key is another separate choice; it does not identify a key to delete.

In the Java producer, serialization runs on the thread calling send(), before the record enters the accumulator. A slow serializer therefore slows the caller even if the broker is healthy. The producer’s max.block.ms setting does not bound time spent in user serialization code.

The diagram distinguishes an encoding failure from a failure after delivery has begun:

A genuine serialization failure means this attempt did not produce a buffered record for delivery. It is different from a network timeout, where a broker may have stored the record before the producer lost the response. It also says nothing about whether an earlier application attempt published the same event.

Handle both exceptions from the producer call and failures its future or callback reports. Diagnose the underlying cause instead of assuming every send failure is a broker outage. Retrying an unsupported value with the same serializer will not repair its encoding, and increasing broker retry settings will not make the value valid.

Error reporting should retain enough context to identify the event and failing serializer without dumping sensitive payloads into logs. If the application must recover failed events, the application needs durable storage or another explicit recovery mechanism. A serializer should not hide failures or create its own uncontrolled retry loop.

Finally, a serializer change affects more than new code. Consumers may replay records producers wrote months earlier. Switching from JSON to another format on the same topic leaves old and new byte representations together unless a migration deliberately handles both.

Even two tools that both call themselves JSON serializers can differ in their output: plain JSON, additional framing bytes, type information, or schema identifiers. Our example produces plain UTF-8 JSON with no schema-registry framing. It is compatible with readers expecting that representation, not automatically with every JSON-related client library.

Before changing an established serializer, check the actual bytes and the readers that must interpret them. Stable key encoding, explicit value formats, and checks against retained records make serialization changes much easier to reason about.

Serializers define how application keys and values become bytes. Producers and consumers need to agree on those bytes and their meaning, even when they use different languages or libraries.

Built-in serializers cover common types; a custom serializer can encode an event object into a defined format such as UTF-8 JSON. Keep key encoding stable, preserve the distinction between null and empty values, and report encoding failures rather than substituting misleading data. Testing the bytes directly helps catch format problems before they become publishing or consumer failures.