An order-tracking application needs new events to appear quickly. An analytics pipeline needs to move a large volume of events efficiently. Both use Kafka, but a setting that helps one workload may make the other worse.

Even within one application, the right choice depends on traffic. A short batching delay may make a quiet order wait longer, yet help a busy system finish orders sooner by reducing request overhead.

In this chapter, we’ll look at how to choose a throughput and latency target, decide where waiting is useful, and compare tuning changes under realistic load. The goal is to find a setting that meets the application’s requirements with room to handle variation.

“Make Kafka faster” leaves too much open. A useful target says how much work must finish, how quickly it must finish, and what successful completion means.

Suppose an order service writes OrderStatusChanged events to orders.status, with order IDs as keys. A tracking consumer applies each event to a database that serves the customer’s order page. The team wants the database update to finish within 100 milliseconds for at least 99% of events while sustaining a peak of 20,000 events per second.

That is a service-level objective, or SLO: a measurable target for service behavior. Here, latency starts when the application submits an event to the producer and ends when the tracking database update completes. It excludes work before submission and any later browser refresh.

The 99th-percentile latency, or p99, marks the duration within which 99% of measured events finish. The team must also track errors and missing completions. A p99 that counts only successful events does not establish that 99% of all submitted events met the deadline.

The workload matters as much as the target. For this example, assume:

At peak, the serialized event stream is approximately 20 MB/s before compression, using decimal megabytes. This is a description of the example workload, not a capacity estimate for twelve partitions.

The team needs two kinds of evidence: latency at the required arrival rate, and the highest rate that still meets the objective without a growing backlog. A setting that wins at light load may lose when requests start competing for resources.

A latency budget divides the available time among the stages that must finish. It helps prevent one stage from consuming time that another stage needs.

For the 100-millisecond target, the team might begin with the following planning allowances. Each allowance includes the work and waiting within that stage; the unused margin covers variation across the path.

The allowances total 80 milliseconds, leaving 20 milliseconds of margin. They are starting points for investigation, not Kafka guarantees or measured stage percentiles. Adding each stage’s p99 does not calculate the pipeline’s p99; different events can be slow at different stages. Measure the end-to-end distribution directly.

The diagram follows an event toward consumer processing. A producer acknowledgment is a separate observation: a fast acknowledgment does not prove that the tracking database is current.

Budgeting also reveals waits that are easy to miss. If the consumer application collects updates for 200 milliseconds before writing them, the 100-millisecond target is already unrealistic for many events. Reducing a producer delay by 2 milliseconds will not resolve that mismatch.

Start with actual event timings. If most of the delay comes from waiting for database connections, tune that path first. If the database finishes promptly but events spend time accumulating in the producer, batching becomes a reasonable place to experiment.

Kafka gains efficiency by moving records in batches. Each request has overhead, so sending more records together can reduce the work needed per event. The tuning decision is how much waiting, if any, is worth that reduction.

For the Java producer, linger.ms permits a delay while records accumulate, and batch.size influences batch size in bytes. A full batch can become ready before the linger interval ends. Neither setting bounds total delivery latency; scheduling, unavailable connections, broker pressure, and retries can add time.

Batches form within each producer and partition. A high topic-wide rate does not guarantee full batches if many producer instances spread their records widely.

Consider one producer writing 100 events per second to one partition, roughly one every 10 milliseconds. With a 5-millisecond linger allowance, an isolated event may wait without gaining a companion. At 10,000 events per second to that same producer-partition pair, about 50 further arrivals fit into 5 milliseconds, assuming an even arrival pattern. The batch may also fill sooner.

These are simplified arrival examples. Real arrivals are uneven, and a zero linger setting does not prevent records from batching while an earlier request is in progress.

The diagram shows why the same tuning change can have different effects. It assumes the longer wait actually produces fuller batches in the busy case.

Intentional waiting is visible in a setting. Queueing is the wait that develops when work competes for capacity. Eliminating the first can increase the second if it creates many small requests.

The reverse is also possible: longer waits can add delay without saving enough work. Once batches are already efficient, additional accumulation time may provide little benefit. Test modest changes and check whether batch sizes and request pressure change in the way you expected.

Producer batching is only one source of deliberate delay. A consumer can ask a broker to accumulate data before replying to a fetch request, and the application can batch work after receiving records.

In the Java consumer, fetch.min.bytes influences how much data the broker waits to gather for a fetch response. For local log reads, fetch.max.wait.ms limits that accumulation wait when insufficient data is available. These settings apply to fetch requests, not separately to each business event.

For example, consider a consumer requesting a 64 KiB minimum with a 25-millisecond maximum wait. If enough eligible data is already available, the broker can respond without spending the full interval waiting for more. With sparse traffic, a small amount of available data may wait until the interval expires. The response can contain less than the requested minimum.

Those values illustrate the mechanism; they are not recommendations. The maximum fetch wait is not a bound on end-to-end latency. Request queueing, network transit, consumer scheduling, and processing still take time.

Larger fetch responses can reduce repeated request work. However, waiting for them spends part of the same latency budget the producer already uses. Independently choosing a “small” delay at every stage can create a large combined delay.

Application batching deserves the same scrutiny. Writing several compatible updates in one database operation may reduce round trips. Waiting too long to collect those updates may make a quiet event stale. An application often needs both a size trigger and a time trigger so that a partly filled batch eventually runs.

Keep fetch size and processing size distinct. Java consumer max.poll.records limits records that one poll() call returns; it does not directly limit the underlying fetch. Changing it can alter how much work the application receives at once, but it does not make each database update faster.

The useful unit of optimization is completed application work. If larger batches only move the backlog from Kafka into a consumer’s memory, throughput has not improved at the boundary the user cares about.

A pipeline usually has several operating regions. At light load, individual processing costs and deliberate waits dominate. As load increases, resources stay busier and batching can become more efficient. Near saturation, queues grow and slow events become much slower.

The point where latency starts rising rapidly is the knee of the load curve. It is a useful observation, not a fixed utilization threshold. The location changes with record sizes, traffic distribution, shared workloads, and hardware.

For the tracking application, the relevant limit is the highest sustained rate that still meets the 100-millisecond objective and its error requirements. A higher rate that completes records eventually but leaves them waiting too long is outside the acceptable operating range.

Find that range by increasing offered load in steps. Offered load is the work the test tries to introduce per second. Compare it with successful completion throughput and with the amount of unfinished work.

At each step, hold the rate long enough to see whether latency and backlog settle. A short burst can look successful while buffers absorb work that the system has not finished. Stop increasing load when the objective fails or unfinished work keeps growing; record why that step failed.

Concurrency can move the limit when too little work is in progress to use available capacity. The Java producer supports asynchronous sends, allowing records to overlap instead of making the application wait after every submission. Applications still need bounded outstanding work and completion handling.

Beyond the useful concurrency level, more outstanding work mainly creates a longer queue. Likewise, larger buffers give waiting work more space but do not necessarily increase the rate at which it finishes.

Compression is another experiment worth considering when byte transfer is expensive. It trades CPU work for fewer bytes. The result depends on the payload and available resources, so compare successful throughput and latency together rather than choosing solely by compression ratio.

Suppose measurements show that the tracking pipeline spends substantial time handling small produce requests at peak load. The team chooses to vary producer linger while holding batch size, compression, consumer settings, resources, and application code fixed.

The prediction is specific: a short wait should create fuller batches, reduce request overhead, and lower queueing at peak. At light load, it may increase latency.

The following results are hypothetical teaching data, not measurements from a Kafka benchmark. Each candidate receives the same event sequence and undergoes the same load steps. Assume repeated runs show similar results, with no errors or growing backlog at the rates the table lists as passing.

The zero-wait candidate gives the lowest latency at light load but fails at the required peak. Suppose its peak throughput still matches arrivals: the problem is excessive waiting, even without a continuously growing backlog.

The 5-millisecond candidate meets the peak requirement and raises the highest passing tested rate substantially. Supporting measurements should show the predicted improvement in batch sizes and request pressure. Without that evidence, the table shows an outcome but does not explain its cause.

Moving from 5 to 20 milliseconds buys only another 1,000 events per second in the passing load steps and a modest peak-latency improvement. It adds 15 milliseconds to the light-load p99. For this tracking application, 5 milliseconds is a reasonable candidate because it meets the stated peak while preserving quicker updates during quiet periods.

An analytics workload with more relaxed freshness requirements might value the additional throughput or lower resource use differently. The decision follows the application’s objective; the table does not establish a universal linger value.

The 25,000-event passing result also does not prove that a 20,000-event workload will meet its target during a broker failure. It establishes healthy-cluster evidence under the tested conditions. Recovery and competing traffic need their own validation.

The diagram summarizes the experiment as a repeatable loop:

Changing one factor makes the result easier to explain. After finding useful individual changes, test them together, because batching, compression, and concurrency interact.

A tuning comparison is only as useful as its measurement method. Keep the event-size distribution, key distribution, producer count, consumer count, and downstream work representative. A single producer sending identical values to one partition may batch and compress far better than the application you intend to run.

Measure event latency from the declared submission boundary through business completion. A stable event ID lets the test match submissions with completed updates and recognize repeated attempts. Record acknowledgment latency separately so you can distinguish publication delays from consumer-side delays.

If the start and finish timestamps come from different machines, account for clock differences. A clock error of several milliseconds can overwhelm the improvement you are trying to measure. Also report the sample count and measurement interval: a p99 based on very few events is a weak basis for a tuning decision.

Watch how the load generator behaves when the system slows down. If it waits for one event to finish before scheduling the next, it automatically reduces offered load during a stall. A real order stream may keep arriving. The test can therefore miss the queueing those additional arrivals would experience.

For a fixed-arrival-rate test, schedule arrivals independently of completions within defined memory and concurrency limits. Record the gap between intended arrival and actual submission if the generator cannot keep up. Keep that delay visible alongside the declared application latency, rather than silently shrinking the workload or starting every timer after the wait.

Do not discard timeouts, failed sends, or events still unfinished when the measurement interval ends. Account for them and observe the remaining work during a bounded drain period. Otherwise, the slowest events disappear from the report.

Finally, compare quiet traffic, sustained peak traffic, and bursts. Let connections, runtime warm-up, cache behavior, and storage writeback settle for the steady-state comparison. Then test startup and recovery separately, since those conditions matter even when they differ from steady state.

Performance comparisons need the same definition of success. Changing acknowledgment requirements, removing TLS, or skipping database work creates a different workload with different properties.

In our example, acks=all waits for the in-sync replicas to receive a write. It does not confirm the tracking database update. Acknowledging sooner by weakening that requirement changes the durability trade-off; it is not an equivalent way to satisfy the original objective.

Application changes need the same care. If batching database updates improves throughput, advance committed offsets only past work that has safely finished. A crash after a database update but before the offset commit can repeat that update. The consumer must preserve its idempotent behavior when processing larger batches.

Timeouts are also failure-policy choices. Shortening a timeout does not make the operation complete sooner; it makes the application stop waiting sooner and may produce more failed or uncertain outcomes. A timeout does not always prove that Kafka never accepted a write. Counting those outcomes separately prevents a misleading latency improvement.

Before adopting a candidate, test a relevant disruption, such as a broker restart, consumer reassignment, or a replay competing with live traffic. Define whether the usual latency objective applies during that disruption and how quickly the pipeline must recover afterward. Spare capacity must serve both new events and delayed work.

Introduce the change gradually and compare the same completion rate, latency distribution, and error measures the experiment used. Keep the previous configuration available if production traffic behaves differently. Once the pipeline meets the requirements with enough spare capacity, further tuning should have a concrete benefit, such as reducing resource cost or supporting a larger expected peak.

Tune for a defined completion rate and latency objective under a representative workload. A small batching wait can reduce overhead and improve latency under load, while a quiet event may experience only the added delay. Producer, fetch, and application waits all consume the same end-to-end budget.

Compare candidates at equal offered load, then find their highest sustained rate that still meets the requirements. Track failures and unfinished work, keep correctness and security settings consistent, and validate bursts and recovery before treating a result as production capacity.