A customer can place, update, and cancel an order. If an application needs to apply those events in sequence, it needs them to reach the same partition. At the same time, events for unrelated orders should be able to spread across partitions so the system can process them in parallel.

A record key helps express which events belong together. In this chapter, we’ll follow how a producer uses that key to select a partition, examine the trade-offs in choosing a key, and see why changes to serialization or partition count can change where records go.

A record key is a value Kafka stores separately from the record’s main value. It often identifies the entity an event concerns. Partitioning is the process of selecting which partition of a topic receives a record.

Consider an orders.events topic that carries several kinds of order-lifecycle events. The contract identifies each event type in the value. A cancellation record might have this destination and key:

Its value describes the cancellation:

The earlier OrderPlaced event would have a different eventId but the same key, ord-1042. The event ID identifies one business event; the order ID groups events about the same order.

Setting orderId inside the JSON does not set the Kafka key. The producer must supply the key separately. Kafka’s built-in routing does not inspect the JSON to discover which order it describes.

Likewise, using the same key does not replace an earlier record or prevent duplicates. Both the placement and cancellation remain separate records with their own positions in the log. The key provides an input to routing, not a uniqueness constraint.

Partition selection happens in the producer client. The broker receives a write for a particular topic partition; it does not examine the order ID and redistribute the record across the topic.

For the concrete behavior here, we’ll use the Apache Kafka Java producer from Kafka 4.3. Other clients may use different defaults, so the routing behavior is a client choice rather than a universal rule Kafka brokers enforce.

The Java producer first checks whether the application supplied an explicit partition. If it did, that choice takes precedence. Otherwise, a configured custom partitioner, a component that implements a routing rule, gets to choose. With neither of those overrides, the producer uses its built-in logic.

The diagram shows that decision order. “Usable key” means the serialized key is non-null and partitioner.ignore.keys is false.

This explains why simply seeing a non-null key does not prove that the producer used it for routing. An explicit partition, custom partitioner, or key-ignoring configuration can select a different destination. An explicit partition must also exist in the topic; supplying an invalid number does not create it.

For our keyed examples, assume no explicit partition, no custom partitioner, and partitioner.ignore.keys=false. All producers use the same key serialization and agree on the topic’s partition count.

Records Without a Routing Key

When the Java producer has no usable key for routing, its built-in logic uses sticky partitioning. It keeps sending records to a selected partition for a period of time to help form efficient batches, then changes the selection. It does not necessarily rotate through partitions once per record.

The built-in routing logic for records without a usable key can also adapt to broker performance. Its purpose is efficient distribution, not keeping every record from one producer instance together permanently. A short run may place several records in the same partition without indicating a routing problem.

If partitioner.ignore.keys=true, a key can still be present in the stored record while this built-in path ignores it for partition selection. A custom partitioner has its own rules; that setting does not control it.

Null and empty keys are different. With a string serializer, an empty string becomes a non-null, zero-length byte array. The producer hashes it like any other non-null key. Sending an empty string for every missing order ID therefore sends all those records to one partition under unchanged key-based routing.

Before routing, the producer serializes the key: it converts the application value into bytes. The built-in keyed route hashes those bytes and reduces the result to a valid partition number.

A hash function always converts the same input bytes into the same number. For a fixed partition count, identical key bytes produce the same destination. Different keys can still select the same partition; there are usually far more keys than partitions.

The Java producer’s built-in calculation is:

murmur2 is the hash algorithm the producer uses here. The bit mask makes the result non-negative, and % takes the remainder after division by the partition count. Do not substitute a language’s ordinary string hash or an absolute-value operation when checking compatibility; they can produce different results.

For the UTF-8 bytes of ord-1042, the non-negative hash is 709621863. With three partitions:

Every record with those same key bytes selects partition 0 under our assumptions. The event type, JSON value, and number of consumer instances do not enter this calculation.

Now suppose one producer sends three events in the order shown below and waits for each send to succeed before sending the next. The topic is fresh, has three partitions, and has no other writers. The Java keyed calculation maps ord-1042 to partition 0 and ord-1043 to partition 1.

The two events for ord-1042 share a partition, while a consumer can process the other order independently. Partition 2 being empty is expected for this small sample. Hashing does not promise that a handful of records will fill every partition evenly.

Nor does a different key guarantee a different destination. With the same calculation and three partitions, ord-1044 also maps to partition 0. A consumer that reads that partition reads records for both orders and uses the keys or values to distinguish them.

Choose a key by identifying the scope of related work. For an application that applies an order’s lifecycle in sequence, orderId is a natural choice. It keeps that order’s events together without requiring unrelated orders to share one destination.

Other keys express different requirements:

None of these is universally correct. If an application must maintain a customer-wide sequence, choosing orderId merely because orders are the event subject may be too narrow. If it only needs per-order sequencing, customerId may group more work than necessary.

A useful key usually has many distinct values and a workload that spreads reasonably evenly across them. The number of distinct values is its cardinality. Low cardinality, such as a key that is always PLACED or CANCELLED, leaves little opportunity to spread traffic.

High cardinality alone is not enough. A topic may contain millions of customer IDs while one customer produces most of the records. That customer’s partition can become a hot partition, carrying much more work than the others. Hashing spreads distinct keys; it cannot divide one key’s traffic while also keeping that key on one partition.

Changing an order key to a random value on every event may spread the load, but it also removes the grouping the application relied on. Before making such a change, decide whether the processing requirement allows you to separate those events.

For a multi-tenant application, identifiers may only be unique within a tenant. If two tenants can both have order 1042, a composite key such as tenant-17:ord-1042 can express the complete identity. Define an unambiguous encoding: if components can contain the separator, escape it or use a structured format.

The key should also remain stable for the entity’s lifetime. An order ID is usually a better grouping value than a mutable customer email address. When the key changes, subsequent records may move to another partition even if the application still considers them related.

Two applications can display the same identifier while producing different key bytes. Serializing a numeric order ID as a binary integer produces different bytes from encoding the same number as text. Likewise, a string key ord-1042 differs from JSON-encoding that string with quotation marks around it.

Those differences matter because the hash operates on bytes. Case changes, added whitespace, character encoding, and changes to a structured key’s serialization can all change the destination.

Producers that must keep related records together need a shared routing contract: the definition of the key, its byte encoding, the partition-selection algorithm, and the partition count that algorithm uses. Matching the field name orderId is only part of that agreement.

This matters when a system uses clients in several languages. Two clients can send compatible Kafka requests while using different partitioning defaults. Verify that they produce the same key bytes and partition numbers for representative identifiers rather than assuming clients that communicate with Kafka also route keys identically.

A practical compatibility check includes normal identifiers, any supported non-ASCII characters, composite keys, and the treatment of missing keys. For example, under our UTF-8 and three-partition contract, ord-1042 must map to 0 and ord-1043 to 1. These small checks can catch an accidental serializer or client-configuration change before it separates related events.

If a required order ID is missing, silently substituting an empty string or a random ID changes the routing meaning. The application should handle that invalid event deliberately, using its agreed error-handling path.

Partition affinity means related records consistently reach the same partition. It provides a place where Kafka can preserve their append order, but it does not establish the intended business sequence by itself.

Our example has placement before cancellation because one producer sends them in that sequence and waits for each send to succeed. If independent producers publish conflicting updates concurrently, hashing their keys cannot decide which business action should come first. Kafka records the append order it receives; the applications must coordinate any stronger sequencing requirement.

Producer retry behavior matters too. Suitable idempotent-producer settings protect against retry-related duplicates and reordering within their supported scope. A stable key alone provides neither protection. At the consumer, processing the records concurrently can also cause a later operation to finish before an earlier one.

An ordinary consumer-group rebalance changes which consumer reads a partition. It does not change the key-to-partition calculation. Similarly, moving a partition’s leader to another broker changes the broker address the producer uses, not the partition number selected for the key.

If the selected partition is temporarily unavailable, the normal keyed route does not simply send that record to a different partition to keep traffic moving. The send may wait, retry, or fail according to client settings and the failure. Redirecting related records elsewhere would break the stable placement the key is meant to provide.

A key also does not directly identify a consumer instance. Group assignments can change, so the same partition—and the records for the same order—may have a different consumer after a rebalance.

The partition count is an input to hash-based routing. Increasing it can change the destination even when the key and serializer remain identical.

For ord-1042, increasing orders.events from three partitions to five changes the calculation:

Existing records stay in partition 0. Once a producer knows about the five-partition layout, new records with that key go to partition 3 under the same keyed algorithm.

The diagram shows the two mappings. It does not show records moving from one partition to another.

The order’s history now spans two partitions, which consumers can read independently. Finishing the old partition’s records before handling the new partition’s records is a sequence the application must enforce during migration; Kafka does not impose that sequence across the two logs.

Producers may also discover the expanded layout at different times. During that transition, producers using different partition counts can select different destinations for the same key. A change that affects routing therefore needs coordination when consumers rely on per-key sequencing or partition-local state.

The same principle applies to changing the hash algorithm, serializer, or explicit routing rules. Stable placement depends on keeping the entire mapping consistent, not just preserving a familiar-looking key string.

Keys let producers express which records should stay together. With consistent serialized key bytes, the same keyed routing algorithm, and an unchanged partition count, related records select the same partition. Explicit partition choices and custom or unkeyed routing follow different rules.

Choose the key around the application’s required scope of related work, then consider how traffic spreads across its values. Treat serialization and partition-count changes as routing changes, and preserve the producer and consumer behavior needed for correct processing. Sharing a key provides stable placement under those conditions; it does not by itself establish business order or prevent duplicate work.