An order service can call a producer’s send() method and continue working before Kafka has stored the event. Between that call and a confirmed write, the producer converts data into bytes, selects a partition, collects records into batches, and exchanges requests with brokers.

Understanding this path helps explain behavior that can otherwise be surprising: a send that returns quickly but fails later, a slow broker that eventually slows down application threads, or an event that reaches Kafka even though the producer reports a timeout.

In this chapter, we’ll follow an order event through those steps. We’ll use the Apache Kafka Java client 4.3.1 as the concrete implementation and assume ordinary, non-transactional sends. Other clients follow similar ideas, but their threading, buffering, and completion APIs can differ.

The producer runs inside your application process. It is a client library with its own memory buffers, metadata cache, network connections, and background work.

Why does it need all of that? If an order service sent one network request and waited for a response for every event, network round trips would limit how quickly it could publish. The producer separates preparing a record from delivering it so that the application can submit more records while earlier ones are traveling to Kafka.

In the Java client, application threads call send(). The producer prepares each record and places it in a record accumulator, an in-memory component that collects pending records into batches for individual topic partitions. A background sender thread takes ready batches and handles network delivery.

The diagram shows this handoff. Metadata supports both partition selection and finding the broker that currently leads that partition.

The important boundary is the accumulator. Once a record has entered it, network delivery can proceed independently of the thread that called send(). The record has not necessarily reached a broker yet.

This diagram leaves out optional extensions such as producer interceptors, which can inspect or modify records before serialization. They do not change the basic separation between preparing records and delivering them.

Suppose the order service publishes to orders.placed. It creates a record with the key ord-1042 and this JSON value:

The amount is in paise, so this event represents an order worth INR 1,299.00. The application might also attach a header named source with the UTF-8 bytes of order-service. Headers carry additional metadata separately from the key and value.

At this point, the application has chosen the topic and event contents. It has not assigned an offset. Kafka assigns the offset when the partition leader appends the record.

Serialization

Kafka transports keys and values as bytes. A serializer converts the application’s key or value into that byte representation.

If the service supplies both the key and the JSON document as Java strings, StringSerializer can encode them. It does not validate the JSON or check whether the amount is correct. If the service supplies an order object instead, it needs a serializer that understands that object’s representation.

Serialization runs on the calling application thread in this client. An expensive serializer can therefore make send() slow even when the brokers are healthy. A value the serializer cannot encode fails before it enters the accumulator; broker retries cannot repair it.

Partition Selection

Before buffering the record, the producer needs to establish its destination partition. The application can specify a partition explicitly or let the client select one.

With the standard Java client configuration, the producer uses a non-null serialized key to choose a partition when the application has not specified one. Keeping the serialized key, partition count, and routing behavior unchanged keeps that key’s records on the same partition. Records without keys use the client’s batching-oriented selection behavior; do not assume they alternate through partitions one record at a time.

For this walkthrough, assume ord-1042 routes to partition 0. This is an example assignment, not a calculated hash result. The producer will add the record to a batch for orders.placed, partition 0.

Selecting the same partition gives related records a common log where Kafka can preserve append order. It does not, by itself, establish business order between concurrent application threads or guarantee that consumers finish processing events in that order.

Choosing partition 0 still leaves a network question: which broker should receive the record?

The producer starts with bootstrap.servers, a list of broker addresses it uses to discover the cluster. It obtains metadata that includes the topic’s partitions, their leaders, and broker connection addresses, then caches that information.

Suppose the producer first contacts Broker 2 and learns that Broker 1 leads orders.placed partition 0. It sends that partition’s records directly to Broker 1. Broker 2 does not forward every write, and the KRaft controller is not in the record delivery path.

The cached information can become stale. If leadership moves to Broker 3, the producer must learn the new assignment and direct subsequent requests there. Refreshing metadata is part of the client’s recovery work; the application does not normally maintain its own partition-to-broker map.

Discovery also explains why a successful connection to one bootstrap address does not prove that publishing will work. The producer must be able to reach the broker addresses the metadata response returns, including the leader it actually needs.

When the producer lacks usable topic metadata, send() may wait for it. Asynchronous delivery does not remove the need to establish where a record belongs before accepting it into the buffer.

The accumulator groups records by topic partition. A batch for orders.placed partition 0 cannot also contain records for partition 1, even if both partitions have the same leader.

This distinction matters because a batch and a network request are different units. A batch contains records for one partition. A Produce request can carry batches for multiple partitions whose leader is the same broker.

Suppose the producer has pending records for three partitions of orders.placed. Broker 1 leads partitions 0 and 2, while Broker 2 leads partition 1. The sender can organize ready batches like this, subject to request size limits:

The two batches the sender sends to Broker 1 remain separate. Sharing a request does not combine their logs, give them shared offsets, or make their writes atomic across partitions.

Batching spreads request overhead across multiple records. When you enable compression, the producer compresses records in batches, which can also reduce network traffic for similar event payloads.

Two settings influence how records accumulate. batch.size controls the usual batch size in bytes, while linger.ms allows a short wait for more records to join a batch. A full batch can become ready before that wait ends. A batch does not have to fill before the producer can send it.

Neither setting promises an exact delivery time. A ready batch might still wait for a connection, an available request slot, or broker recovery. Likewise, increasing batch capacity does not ensure that a low-volume partition will produce large batches. The arrival rate for that particular partition matters.

After transmission, a request is in flight while its outcome remains unresolved. The sender can manage multiple requests without requiring the application to wait after every record. Records associated with unfinished work still need memory; handing bytes to the network does not immediately finish the producer’s responsibility for them.

For a record it successfully adds to the accumulator, Java’s send() returns a Future<RecordMetadata>. A future represents an outcome that may become available later. The application can also supply a callback, a function the producer invokes when the send succeeds or fails.

Returning a future is not a broker acknowledgment. It gives the application a way to observe the eventual result, which may also be a failure the producer discovers during preparation.

Let’s follow evt-7001 through a successful send. Assume the producer uses acks=all, the topic has enough in-sync replicas to accept the write, and partition 0 assigns this record offset 42. In-sync replicas are the replicas Kafka considers sufficiently caught up with the leader.

The sequence separates the application’s initial call from the later delivery result:

On success, record metadata identifies orders.placed, partition 0, offset 42. The broker response supplies the information the client needs to determine each record’s offset within the batch.

What success confirms depends on acks. With acks=all, the leader waits for the required replication acknowledgment, subject to the topic’s minimum in-sync replica requirement. With acks=1, it acknowledges its own append without waiting for followers. With acks=0, the producer does not wait for a broker acknowledgment, so successful client completion does not confirm broker storage and the client returns no broker-assigned offset.

None of these outcomes confirms that a reporting consumer has read the event, committed its position, or updated a database.

For ordinary asynchronous sends, callbacks generally run on the producer’s I/O thread. A callback that makes a slow database call can delay other producer work. Keep callbacks brief, and hand off expensive work to an application-managed mechanism with bounded capacity. The producer can report some failures on the calling thread, so callback code should not depend on always running in the background.

An application must handle both errors send() throws and failures its future or callback reports. Ignoring both completion mechanisms means a service can keep accepting orders while missing publishing failures.

The accumulator can absorb a temporary burst, but it cannot absorb a sustained mismatch between production and delivery rates.

Suppose the service submits 20,000 events per second while the producer can deliver only 12,000. Pending work grows by roughly 8,000 events per second for as long as those rates continue. The number of seconds before memory fills depends on record sizes, compression, and other producer activity.

The buffer.memory setting limits the memory available for buffering records. It is not a hard limit on the producer’s total memory use; requests, compression, and other structures also consume memory.

When buffer space runs out, application threads trying to submit more records may wait. This is backpressure: a slower downstream component causes upstream work to slow down. If space does not become available within the allowed wait, the send fails.

Several timing settings apply to different parts of this process:

These are not interchangeable deadlines. In particular, max.block.ms does not include time spent in user serializers or a custom partitioner. The delivery timeout is also not a complete deadline for an order service’s business operation. Records joining an older batch can share its earlier expiration time.

Adding more buffer memory can help with a short burst. Under sustained overload, it mainly allows a larger backlog to build before the application feels the slowdown. The service still needs a policy for slowing incoming work, rejecting it, or retaining it durably for later publishing.

The producer’s buffer is in memory. If the application process crashes before buffered events reach Kafka, the producer cannot recover those events from its old buffer after restarting. To recover accepted business events after a process crash, the application must store them durably.

Different failures require different responses. A temporary connection loss may clear on retry. An invalid serialized value or missing write permission usually needs a change to the data or configuration.

For retryable failures, the sender can retain the affected batch, wait before retrying, refresh metadata when necessary, and make another attempt within the configured limits. The application observes the final result rather than receiving a separate failure notification for every internal attempt.

The difficult case is an uncertain outcome. Suppose the leader appends evt-7001 at offset 42, but the producer never receives its successful response. The producer knows it did not receive confirmation. It does not know from that fact alone whether the write happened.

Without duplicate protection, retrying that write can create another record. An idempotent producer uses producer identity and sequence information so brokers can recognize retries of the same send. This protects against duplicates the producer’s own retries could introduce; it does not deduplicate a fresh application send because its JSON contains the same eventId.

Consequently, do not automatically interpret a delivery timeout as “Kafka has no copy.” Blindly calling send() again can turn uncertainty into a duplicate business event. The application needs a recovery policy that accounts for what it can establish about the previous attempt.

Retries also affect ordering. With idempotence disabled and multiple requests in flight on a connection, the producer can retry a failed earlier batch after a later batch succeeds, changing their order in the partition log. Idempotence preserves ordering with its supported settings. This remains a partition-level property, not a global ordering guarantee across the topic.

The useful architectural distinction is between work the client can retry internally and decisions that require application knowledge. The producer can reconnect to a broker. It cannot decide whether publishing another OrderPlaced event is correct for the business.

The Java KafkaProducer is thread-safe and supports sharing across application threads. Reusing an instance lets those threads share buffers, metadata, and connections. Creating one producer per order repeatedly pays initialization costs and gives each instance fewer records to batch.

A long-lived service should therefore give the producer a clear owner that creates it, makes it available to publishing code, and closes it during shutdown. Separate instances can be appropriate when workloads need different configurations or isolation, but they should be a deliberate choice.

Calling get() on every send’s future before submitting the next record makes the application wait one record at a time. That can be useful when the application needs an immediate result, but it reduces the opportunity to overlap sends. Asynchronous submission is useful only if the application still tracks failures and limits outstanding work.

The flush() method makes buffered records eligible for immediate sending and waits for previously submitted records to complete. Completion can mean success or failure, so the application must still inspect delivery results. Flushing after every event largely defeats the purpose of accumulating records into batches.

For an orderly shutdown, first stop accepting new publishing work and let active application threads finish submitting their records. Then close the producer with enough time for pending delivery to finish, while accounting for failed sends. If the shutdown deadline expires or the operating system forcibly stops the process, you may not know the outcome of pending records, and Kafka may never receive some of them.

For the order service, shutdown is complete only when it has accounted for its pending events according to its recovery policy. An empty application request queue alone does not tell it whether the producer has finished delivering them.

The producer prepares records on application threads, buffers them in batches by topic partition, and uses a background sender to deliver requests to partition leaders. Metadata connects the logical destination—a topic and partition—to the broker that currently serves it.

Submitting a record and confirming its delivery are separate steps. Batching and asynchronous I/O improve efficiency, while bounded memory and timeouts limit how much unfinished work the producer can hold. Applications still need to observe delivery results, handle uncertain outcomes, and manage the producer’s lifetime. A successful produce acknowledgment confirms the configured write condition, not the completion of downstream business work.