An order service publishes events to Kafka, and a fulfillment application uses them to track each order. At first, the producer uses a random event ID as the record key. Traffic spreads across partitions, but an order’s cancellation can reach one consumer while its placement is still waiting in another partition.

The team changes the key to the customer ID. Related events now stay together, but one large customer generates enough traffic to slow down an entire partition, including unrelated customers whose keys route there.

Both choices are valid Kafka keys. Neither fully matches this application’s needs. In this chapter, we’ll work through how to choose a key around processing requirements, evaluate its load distribution, and keep that decision safe as producers and consumers evolve.

Start by identifying which operations the application must apply in sequence. For fulfillment, that might be all lifecycle changes for one order. Changes to different orders can proceed independently, even when the orders belong to the same customer.

This gives us an ordering boundary: the set of related events whose sequence matters to the application. A useful starting point is the smallest stable business identity that contains that set. Here, it is orderId.

Our example uses an orders.events topic containing order-lifecycle events. Assume the order service establishes their business sequence and publishes them in that sequence, with producer settings that preserve it through retries. All producers use the same key encoding and keyed routing rule, and the partition count stays fixed.

For the following illustrative mapping, ord-1042 routes to partition 0 and ord-1043 to partition 1. The offsets show append order within each partition; the diagram omits other partitions.

The fulfillment application can apply the placement and cancellation of ord-1042 in order while processing ord-1043 independently. A partition contains many keys, so this does not mean each order receives a dedicated partition.

The topic boundary matters too. Putting placement in orders.placed and cancellation in orders.cancelled does not create a shared sequence, even if both records have the same key and partition number. They belong to separate logs. Consumers combining those topics need their own coordination rules.

Nor can a key establish business order between independent writers. If two services publish conflicting changes concurrently, Kafka preserves their append order within the partition; it does not determine which change should logically come first. Ownership, sequencing, and consumer behavior must support the ordering promise.

An event usually contains several plausible identifiers. Compare them against the actual processing requirement before choosing one.

For fulfillment, orderId is the best fit among these candidates. For a service that must serialize all changes to one customer account, customerId could be appropriate. The event’s subject provides a starting point, but the state the application maintains determines the required boundary.

Some operations span more than one entity. Choosing one account ID as the key cannot make a transfer between two accounts atomic. That groups only one side of the relationship. Choosing transferId groups the transfer’s events but does not serialize all activity on both accounts. Such requirements need application coordination beyond partition placement.

A key should remain stable for as long as the related history matters. Customer email addresses, order status, and priority are usually poor identities because they can change. A new value can send subsequent events to a different partition.

When records are independent and no consumer needs persistent grouping or keyed state, a null key can be a deliberate choice. For example, independent diagnostic samples may not require per-device order. Document that decision; consumers should not infer grouping from an accidental run of records in the same partition. A missing required order ID is a data error, not a reason to silently switch to unkeyed routing.

The record key is separate from the event value. Including orderId in JSON does not automatically use it for partition selection.

For example, a cancellation record in orders.events uses the key ord-1042 and carries this value:

Our contract requires the producer to encode data.orderId as the Kafka key using UTF-8, without JSON quotation marks or added whitespace. It must reject missing or empty order IDs before publication. The event ID identifies the cancellation; the key groups it with other events for that order.

Stable routing requires agreement on both bytes and behavior. Two clients may display the same identifier but serialize it differently. A plain string, a JSON string, and a binary integer can produce different bytes. Structured keys also need a canonical encoding: one agreed representation for the same logical identity.

For tenant-local order IDs, use the complete identity. A key such as tenant-17:ord-1042 works if the contract forbids the separator inside either component or defines escaping. Without such a rule, different component pairs can produce the same combined string. Two tenants accidentally sharing a key also become indistinguishable to downstream state keyed by that value.

The producer’s routing settings are part of this agreement. In the Apache Kafka 4.3 Java producer, built-in keyed routing uses a hash of the serialized key when no explicit partition or custom partitioner overrides it and partitioner.ignore.keys=false. Check that other clients route the same key bytes to the same partitions. A non-null stored key alone does not prove the producer used it for routing.

Document the key definition, encoding, routing rule, and partition-count assumptions together. For multiple producer implementations, keep shared examples of input identifiers, expected bytes, and expected partitions. Include composite identifiers and any supported non-ASCII characters. These checks are especially valuable during client upgrades or serializer changes.

A key’s cardinality is its number of distinct values. Many distinct order IDs give hashing opportunities to spread work; a few status values do not. But cardinality says little about how much traffic each value produces.

Suppose a marketplace produces 24,000 events per second across 12 partitions. One tenant contributes 12,000 of those events. If the key is tenantId, that tenant alone sends 12,000 events per second to one partition, before accounting for other tenants that hash there.

Assume a measured consumer workload can process 3,000 events per second per partition using the application’s sequential processing model. This is an illustrative measurement, not a Kafka throughput limit. The large tenant’s partition accumulates at least 9,000 events of backlog each second, even if every other partition is keeping up.

Changing the key to the complete order identity can distribute that tenant’s independent orders. Across 12 partitions, 24,000 events per second averages 2,000 per partition, but that average is only a starting point. Actual placement, bursts, and processing costs still determine whether the busiest partition can keep up.

Evaluate candidate keys using representative traffic. For each candidate, serialize and route a sample using the intended producer behavior, then compare records, bytes, and estimated processing work per partition over realistic time windows. Include promotions, bulk imports, and the largest customers. A day-long average can hide a ten-minute overload.

A hot key is one key responsible for disproportionate work. More partitions can spread other keys, but ordinary keyed routing still assigns that hot key to one partition. Adding consumers to a regular consumer group does not split ownership of that partition among them. Application-level concurrency can help independent work, but it must preserve any required sequence for the hot key.

Before changing the key, consider whether batching, cheaper processing, or removing an unnecessary serial dependency can reduce the bottleneck. If the business truly requires one sequential stream and its arrival rate exceeds processing capacity, partitioning alone cannot satisfy both requirements.

A common response to a hot key is salting: adding a suffix that spreads one logical key across several routing keys. For example, tenant-17:0 through tenant-17:7 create eight possible groups.

This can help when the application can split the work and combine the results safely. Counting independent page views may allow partial counts across buckets followed by a combined result. The processing design must still account for retries, duplicates, and how the application merges partial results.

For an order lifecycle, a random suffix per event is unsafe if consumers rely on placement before cancellation. The suffix creates separate logs for operations that were supposed to share a sequence. Adding sequence numbers does not automatically repair this; consumers would need rules for buffering, missing events, and recovery.

A better split follows a genuine independent unit. If a large tenant contains many orders that the application can process independently, key by tenant and order. If the system requires tenant-wide sequencing, that split changes the requirement; confirm that the business accepts the change.

Custom routing can isolate a large workload from smaller ones or reserve capacity for it. It does not make one sequential key parallel. It also introduces a mapping that publishers must share and update safely. Prefer a natural business key when it already provides the required grouping and distribution.

One topic key cannot serve every access pattern equally well. Fulfillment needs order-level grouping, while customer reporting may need to maintain totals for each customer across many orders.

Keep the source topic’s key aligned with its primary contract. A downstream application can read those events and write a derived topic using another key. This is repartitioning: producing records into a new partition layout based on a different grouping.

The diagram shows two uses of the same source. Fulfillment reads order events directly. Reporting first groups them by customer through a separate topic.

The derived stream gives reporting a place to group customer state without changing fulfillment’s source contract. It costs additional storage, network traffic, processing, and recovery work. The application that writes it must handle failures between reading input and producing output.

Repartitioning does not reconstruct a global historical order. Events from separate source partitions can reach the derived partition in different interleavings. Customer reporting must accept that behavior or use explicit business sequencing rules. An order-independent sum has different requirements from enforcing a customer-wide spending limit.

Related stateful inputs also need deliberate alignment. If an application processes two streams together by customer, both streams must use matching key bytes and partition mappings. Otherwise, the processing system must redistribute the records before processing them together. Kafka does not automatically join records because their keys look alike.

The key also affects what remains in a compacted topic. Log compaction can remove older records for the same key within a partition. It is useful for retaining updates to keyed state, but it changes what a historical reader can reconstruct.

For an orders.current-state topic, orderId is a natural key if each value describes the current state of that order. Using customerId would allow updates for different orders belonging to one customer to supersede one another within the partition. Using a unique event ID would prevent successive updates to one order from sharing a compaction identity.

That does not mean an order-lifecycle topic should use event IDs to avoid compaction. If consumers need every retained transition, choose a cleanup policy that preserves that history for the required period. The routing key and retention policy should support the topic’s purpose together.

A mutable key creates another problem for state topics: an update under the new key does not remove state under the old key. Any identity change needs an explicit transition, including deletion of the old identity where appropriate. Do not treat a key change as an ordinary payload edit.

A partitioning strategy must account for growth. Increasing the partition count can change future destinations under hash-based routing. Existing records stay where they are, so one order’s history can span old and new partitions. Producers may also learn about the new count at different times.

Changing the serializer, hash algorithm, or key definition can create the same kind of split. A consumer-group rebalance or broker leader change is different: it changes who handles a partition, not the key’s destination partition under an unchanged routing rule.

For a system that must preserve event order during migration, a controlled move to a new topic can make the transition explicit. The following is one possible approach when a publication pause is acceptable, not a procedure Kafka performs automatically.

The pause must cover every relevant publisher and any queued publication work. Draining means completing the required application effects through the recorded boundaries, not merely fetching records or advancing offsets. If consumers keep state local to partitions, the migration must rebuild or transfer that state into the target layout before live processing relies on it.

Define how consumers will replay retained history after the switch. A live drain does not order old and new topics for a future rebuild. The rebuild may need to finish the old history before consuming the new stream, or start from a verified state snapshot at the cutover boundary.

Also define how to recover if the cutover fails. Once consumers have processed new-topic events, returning publishers to the old topic without reconciling those events can create gaps or duplicate effects. Copying or publishing to both topics needs explicit identity and deduplication rules; it is not automatically a safe migration.

Before launch, verify the strategy with representative keys and traffic, then observe actual destinations and per-partition lag. Keep the ordering requirement, key encoding, largest-key workload, retention purpose, and migration responsibility documented together. These decisions determine how applications will use the event history long after you deploy the first producer.

Choose a key around the smallest stable business identity whose events need coordinated processing. Check that routing distributes its traffic within each partition’s capacity, and make its encoding and routing behavior part of the event contract.

Use derived streams when consumers need different groupings, align compaction with the state the topic represents, and treat changes to keys or partition mapping as migrations. A sound strategy preserves the required relationships between events while allowing independent work to scale.