A Kafka cluster can have spare capacity while its applications struggle to publish or process events. A producer may wait after every send, preventing useful overlap. A consumer may fetch quickly but spend most of its time making individual database calls.

Client tuning starts by finding those limits. Configuration helps when it changes the work that is actually holding the application back. Larger buffers or more threads can just give unfinished work more places to wait.

In this chapter, we’ll work through producer and consumer tuning, connect the important settings to observable behavior, and examine how to improve throughput while preserving safe processing and recovery.

We’ll use Apache Kafka 4.3 Java clients with a KRaft-based cluster. The orders.status topic has twelve partitions and replication factor three. Producers publish non-transactional OrderStatusChanged events with order IDs such as ord-1042 as keys. A regular consumer group, order-tracking, applies them to a database.

Assume the deployment has already configured production authentication and TLS. Producers use idempotence and acks=all, and the topic uses min.insync.replicas=2. The database handles repeated events safely using their event IDs. Consumers read from partition leaders and commit offsets only after the corresponding database work finishes.

The example workload peaks at 20,000 events per second, averaging 1,000 serialized bytes per event. Traffic is reasonably balanced across keys. These numbers describe the application, not an expected capacity for a particular cluster size.

Before changing anything, record the client versions, effective configuration, process counts, resource limits, and event-size distribution. Give client instances distinguishable client.id values so you can distinguish their measurements. Record both producer acknowledgment throughput and completed database updates per second, along with their latency distributions and errors.

Also measure where the application spends time: serialization, send calls, producer callbacks, deserialization, database calls, and offset commits. A CPU profile can show expensive parsing or object allocation that Kafka settings will not fix.

Keep a saved baseline and change one relevant factor at a time. The configuration fragments below are experiment candidates to merge into an existing working setup. They omit connection addresses, credentials, serializers, and deserializers because those remain application-specific and unchanged. They are not complete client configurations or universal production recommendations.

Start with the way the application uses the producer. KafkaProducer is thread-safe and intended for reuse. Creating one for every event repeatedly pays connection, metadata, buffer, and thread costs. Several application threads can share an instance when their configuration and lifecycle requirements match.

The diagram separates application work from the producer’s background sending work. Batches collect records for individual partitions.

This structure allows new submissions to overlap with earlier requests. A loop that calls send(record).get() before submitting the next record removes much of that opportunity. If each acknowledgment takes 4 milliseconds, one such loop can submit only about 250 records per second, even before accounting for other work.

Use callbacks or retained futures to observe asynchronous completion without waiting after every individual submission. Bound outstanding work and handle both immediate submission exceptions and later delivery failures. Returning from send() does not establish that the record reached Kafka.

Callbacks usually run on the producer’s I/O thread. Keep them short; a database call or expensive log operation there can delay unrelated sends. If completion handling needs a separate executor, bound its queue and define what happens when it fills. Moving work into an unlimited executor queue only relocates the problem.

Avoid calling flush() after every record. It forces buffered records to become eligible for sending and waits for earlier sends to complete, reducing batching opportunities. Use it where the application needs a completion boundary, while still checking individual failures. At shutdown, stop accepting new work and allow pending deliveries to resolve before closing the producer.

Once the application allows overlap, inspect actual batch sizes and request rates. Small batches under substantial traffic may mean that producers spread records across too many producer-partition pairs, or that the producer sends before enough records collect.

batch.size controls the usual per-partition batch allocation in bytes. linger.ms allows time for more records to join. A full batch can become ready earlier; the linger value does not bound total delivery time.

Suppose one producer writes evenly to twelve partitions at 12,000 events per second. Each partition receives about 1,000 per second. In 5 milliseconds, only about five additional events arrive for each partition. With roughly 1,000-byte events, increasing the batch allocation from 16 KiB to 64 KiB does not make those arrivals fill it. This is an arrival estimate, not an exact prediction of batch contents.

If measurements instead show batches repeatedly reaching their size limit, a larger allocation is a more plausible experiment. It can reduce batch overhead at the cost of more memory per active batch. Keep the key strategy intact if the application depends on order-level ordering.

For a workload where batches are filling and network use is significant, the following fragment illustrates a candidate configuration:

Here, 65536 is 64 KiB. Test batch size, linger, and compression separately before evaluating their combination. The fragment records a possible combination, not an instruction to change all three at once.

Compression trades CPU work for fewer transmitted and stored bytes. lz4 is one available codec, not an automatic best choice. Compare it with the existing codec using real event contents. Repeated JSON fields and already compressed payloads behave differently.

Look for the predicted effects: fuller batches, fewer requests for the same successful record rate, or fewer bytes with acceptable CPU and latency. If the producer becomes CPU-bound, investigate serialization, compression, and allocation before adding more buffering. A serializer that performs a remote lookup per event can dominate a producer that otherwise has efficient batches.

Producer buffers absorb differences between submission and delivery rates. They need room for normal bursts, but they cannot make a slow broker or saturated network drain faster.

buffer.memory budgets the producer’s record buffer pool, not the whole process. Compression, requests, application objects, and other allocations also consume memory. Raising it within an unchanged container limit can cause memory pressure elsewhere.

As a rough illustration, 400 active 64 KiB batch buffers represent 25 MiB before other memory costs. That is not a sizing formula: a partition can have multiple batches outstanding, and actual allocations vary. It shows why you must consider batch size and active partition count together.

When submission blocks, distinguish buffer exhaustion from metadata or connectivity problems. The Java producer’s max.block.ms limits waiting for metadata and buffer allocation during send(). It does not cover arbitrary time spent inside user serializers. Increasing the limit allows a longer wait; it does not remove its cause.

Delivery has separate timing controls. request.timeout.ms governs waiting for a request response, while delivery.timeout.ms limits how long the producer has to report delivery success or failure after send() returns. Keep the delivery timeout at least as large as the request timeout plus linger. Choose these limits around acceptable failure behavior rather than treating them as throughput controls.

Another setting, max.in.flight.requests.per.connection, limits unacknowledged requests on each broker connection. It counts requests, not records or application tasks. Restricting it to one can leave useful network overlap unused.

For this Java client version, idempotence supports values up to five while preserving order for the producer’s records within a partition. The following compatible settings make the intended behavior explicit; leave retries enabled:

Idempotence prevents duplicates from the producer’s supported retry mechanism. It does not deduplicate a business event that the application submits again as a new send. Nor does an acknowledgment prove that the tracking database has processed the event.

If timeouts or retries rise after increasing concurrency, inspect broker response time, throttling, and network conditions. More outstanding requests may be adding pressure to a resource that is already the limit.

Consumer tuning has three different units: the fetch response, the records poll() returns, and the work the application completes. Treating them as one batch makes it easy to adjust the wrong setting.

The diagram shows those boundaries. The client can retain fetched data and return it across multiple polls.

Two byte settings shape fetch responses. max.partition.fetch.bytes applies to each partition, while fetch.max.bytes applies to the response to a fetch request. Neither is a hard process-memory limit. Kafka can return an oversized first batch from the first nonempty partition so the consumer can make progress, and a consumer can fetch from multiple brokers concurrently.

Increase a fetch allowance when responses repeatedly reach that limit, data remains available, and the application can process more than it receives. Small responses on a quiet topic are not evidence that the allowance is too small.

fetch.min.bytes and fetch.max.wait.ms control accumulation before a local-log fetch response. Larger minimums may reduce request frequency but add waiting when traffic is sparse. Change them only if request overhead matters and the latency budget permits it.

Memory planning must include decompressed data, deserialized objects, application queues, and database batches. A 1 MiB compressed response can require considerably more memory after decoding. Increasing fetch sizes while application queues are already growing usually makes the consumer hold more unfinished work.

For large records, make sure the producer’s request-size limit and the broker’s or topic’s record-size limit allow the intended payloads. These limits apply to different units, such as whole requests or compressed record batches, so setting every byte limit to the same value is not a reliable design. For unusually large events, first consider whether the event contract can carry a smaller representation without losing necessary information.

max.poll.records limits how many records one Java poll() call returns. It does not directly reduce the client’s fetch size. Its main value for a sequential processing loop is controlling the amount of application work between polls.

Suppose one consumer spends an average of 4 milliseconds applying each event. Processing 500 records takes about 2 seconds before commit time, retries, or other overhead. Returning 100 records reduces the average processing portion to about 400 milliseconds, but the worker still completes only about 250 records per second.

A smaller poll lets the application return to polling sooner and limits how much it processes at once; it does not remove the database cost. Use observed slow cases as well as averages when deciding how much work is safe to accept.

The following fragment illustrates a smaller application batch with manual offset management:

The application must implement successful-processing checks and commits; disabling automatic commits does not implement that logic. The five-minute poll interval shown here is a liveness limit, not a processing-latency target.

max.poll.interval.ms bounds the gap between poll invocations when using group management. If processing prevents timely polling, the consumer can lose its assignment. The precise reassignment timing depends on the group protocol and static membership. Increasing the limit may accommodate legitimately longer work, but also lets this check take longer to detect a stuck processing loop.

Put time limits on downstream operations and bound retries. A poll containing one database call that hangs indefinitely is unsafe regardless of its record count. If database round trips dominate, test compatible batch updates or prepared operations while preserving event ordering and duplicate handling.

Heartbeat settings are separate from the poll interval. Under the newer Consumer group protocol, the broker controls heartbeat and session timing; classic-client heartbeat settings do not tune that protocol. Changing heartbeat frequency is not a way to make application processing faster.

If each consumer does useful work continuously and downstream resources have room, more consumer instances can increase processing capacity. A straightforward design gives each instance one consumer and a sequential processing loop.

With twelve partitions and six group members, a balanced assignment gives each member two partitions. Expanding to twelve members can distribute those partitions across twelve processes. Expanding to eighteen leaves some members without a partition from this topic. More members cannot split one hot partition’s assignment.

An alternative keeps one thread responsible for the consumer and delegates processing to workers. The Java consumer is not generally thread-safe; worker threads should return completion information to its owner rather than call its poll, assignment, or commit methods concurrently.

This design adds responsibilities. Bound queued work and use pause() and resume() from the owner thread to control intake for affected partitions while continuing to poll. Preserve partition order with serial work per partition if the application requires it. Processing within a partition concurrently needs additional ordering and progress tracking.

Consider a consumer that may process independent orders concurrently. In partition 0, all records before offset 42 are finished. Offsets 42 and 44 finish, but 43 is still running. The diagram shows the resulting commit boundary.

The safe commit is 43, not 45. Once 43 finishes, the application can advance to 45 because 44 is already complete. A database update at 44 may run again after a crash before that advance, so idempotency still matters.

For each partition, track how far processing has finished without leaving an earlier record unfinished. In a worker-based design, a no-argument commit can use fetched positions beyond completed work; supply the explicitly tracked safe offsets instead.

Reassignment makes worker management harder. Stop dispatching revoked partitions, stop or drain their old work, and prevent stale completions from advancing the new assignment’s progress. A worker already performing an external update does not stop merely because Kafka changed ownership. The application must handle possible overlap through its ordering, idempotency, or external ownership controls.

Prefer the simpler consumer-per-thread approach when it provides enough parallelism. Introduce a separate worker pool when the measured benefit justifies this additional coordination.

Committing after every record can add a network round trip to otherwise short processing. Committing after a group of completed records spreads that cost, but leaves more work that may repeat after failure.

For a sequential loop with auto-commit disabled, processing a poll fully and then committing its safe next offsets is a simple starting point. If processing fails partway through, do not commit positions past the failure. Recover deliberately by retrying, seeking to safe positions, or restarting from committed progress according to the application’s policy.

commitSync() waits for the commit result. commitAsync() allows the loop to continue, but the application still needs to handle commit errors. Retrying an old offset map after the consumer has committed newer progress can move the saved position backward; do not blindly resubmit stale commits.

Choose how often to commit based on measured commit cost and acceptable repeated work. A lower commit rate can improve throughput when commits are expensive. It can also increase recovery time and the number of duplicate processing attempts. Committing earlier than successful processing creates a different problem: a crash can skip unfinished updates altogether.

Neither commit method makes a Kafka offset and an external database update atomic. Tuning the frequency changes overhead and the replay window, not that underlying boundary.

Use client evidence to choose a small experiment. The same visible symptom can have several causes, so each row below pairs a possible change with something to verify first.

The named metrics are Java client metric names; monitoring exporters may rename them. Read them alongside application timings. For example, time-between-poll-max measures the gap between poll invocations, not just time spent inside Kafka's poll implementation.

Suppose tracking consumers receive records promptly but each update waits for a database connection. Raising fetch limits will not create database capacity. First test whether compatible update batching reduces connection use and round trips. Only add callers if the database can use the extra concurrency.

For every candidate, compare successful publication, completed application work, latency, errors, and memory under the same traffic. Check the busiest partitions and instances, since averages can hide one struggling worker. Run long enough for queues and memory use to settle.

Then test a relevant failure. Confirm that pending producer sends resolve visibly, consumers resume from safe progress, and repeated events do not corrupt state. A useful tuning change improves sustained work while keeping the application’s recovery behavior intact.

Producer tuning starts with reuse, bounded asynchronous sending, and fast completion handling. Adjust batches, compression, buffers, and request overlap only when measurements show how those changes can help.

Consumer tuning separates fetch capacity from poll size and application processing. Add parallelism where work can run independently, track completed progress per partition, and choose commit frequency with recovery cost in mind.

Judge the result by successful completed work, latency, resource use, and behavior during failure. Larger settings are useful only when they address the measured limit.