An order service publishes events successfully, but the reporting dashboard keeps falling behind. The team adds brokers, yet reports do not get any fresher. Kafka has spare capacity. The reporting database does not.

Situations like this are easier to understand when you treat performance as a property of the whole record-processing path. Producers, brokers, replicas, consumers, and external systems each do work, share resources, and make records wait.

In this chapter, we’ll build a mental model for that path. We’ll connect throughput, latency, queues, batching, and partition parallelism, then use a realistic workload to reason about a slowdown before choosing what to change.

Throughput is the amount of work a system completes per unit of time. Latency is the time one operation takes between a defined start and finish. Neither measurement is useful until you say what counts as completion.

Consider an OrderPlaced event in orders.placed, with key ord-1042. The producer sends it, Kafka acknowledges the write, and a reporting consumer eventually updates a database. There are several useful measurements along that path:

Timing only the return of an asynchronous send() call does not measure acknowledgment latency. Similarly, fetching 20,000 records per second says little about reporting capacity if database updates finish at only 8,000 per second.

Record counts and byte counts answer different questions. Small records can make request overhead or application work the limiting factor. Large records can exhaust network or storage bandwidth at a much lower record rate. Always keep record size alongside records per second, and distinguish serialized bytes from compressed bytes.

For latency, look beyond the average. The 99th percentile, or p99, is the latency at or below which 99% of the measured operations fall. A low average can hide a small but important fraction of orders waiting several seconds. Measure failures and timeouts too; dropping them from a report can make an overloaded system appear healthy.

Our running example uses a KRaft-based cluster with ordinary replicated partition logs on broker storage, regular consumer groups, and consumers fetching from leaders. The events are non-transactional. All workload numbers are illustrative, not Kafka capacity claims.

A record moves through several stages, but those stages overlap across records. A producer can prepare the next batch while a broker replicates the previous one and a consumer processes older events.

The diagram shows the main path and two distinct completion points. It shows data movement, with followers and consumers fetching from the leader.

Producer completion and business completion can be far apart. Controllers manage metadata and leadership, but order records do not travel through the controller quorum on this path.

At each stage, separate service time, the time spent doing the work, from queueing time, the time spent waiting for the stage to accept it. A database update might take 2 milliseconds once it starts, yet wait 500 milliseconds for a connection. Making serialization slightly faster will barely affect that event’s end-to-end latency.

Latency also includes network transit, deliberate batching waits, and any retry delays. These costs can overlap, so adding independently measured stage percentiles will not give the pipeline’s p99. To explain a slow event, follow its actual timing through the system. If timestamps come from different machines, account for clock differences.

For a steady workload, completed throughput cannot exceed the capacity of the slowest required stage. That stage is the bottleneck. Its identity depends on the workload: producer CPU may limit a pipeline for one event format, while database writes limit it for another.

The bottleneck can also move. Faster database writes may expose a consumer CPU limit that previously went unnoticed. A useful performance model explains the current limit and predicts what becomes important after it changes.

Suppose the reporting group receives 12,000 events per second, but its consumers and database together can finish only 10,000. Unfinished work grows by 2,000 events every second. After one minute, the pipeline has accumulated 120,000 additional unfinished events.

That work can wait in Kafka’s retained log, a consumer’s fetched records, or an application work queue. Moving it between those locations does not make it complete.

For a sustained overload, the basic relationship is:

Backlog growth per second = arrival rate − completion rate.

This describes unfinished application work. An offset-based lag measurement may describe a different boundary, such as fetched or committed progress. A consumer that fetches eagerly into a large application queue can appear caught up while business processing remains far behind.

Queues can grow even when average arrival rate is below average capacity. Traffic arrives in bursts, and processing times vary. As the workload uses more of the available capacity, there is less room to absorb either variation, so waiting and tail latency often rise sharply before completed throughput stops increasing.

Headroom is the capacity left beyond the current workload. It lets a pipeline absorb bursts and finish delayed work. There is no universal safe utilization percentage: acceptable headroom depends on burst sizes, processing variability, latency requirements, and failure scenarios.

Suppose the reporting pipeline now has a backlog of 120,000 events. Live arrivals settle at 8,000 per second, while completion capacity remains 10,000. Only the spare 2,000 completions per second can reduce the backlog:

Catch-up time ≈ backlog ÷ (completion capacity − live arrival rate).

Here, catch-up takes approximately 60 seconds. Dividing by the full 10,000 would incorrectly predict 12 seconds because it ignores new arrivals. The estimate assumes stable rates, similar work per event, available retained records, and no further pauses. If completion capacity only matches arrivals, the backlog never shrinks.

A larger buffer can absorb a brief burst. It cannot fix a lasting capacity deficit. Eventually, the system must slow incoming work, reject it, time out, or run out of space. Backpressure means making upstream work slow down when downstream capacity is insufficient. Kafka can hold a backlog independently of a producer, so a slow reporting database does not automatically slow order publication.

Capacity is not just a hardware limit. It also depends on how much useful work each request carries.

A request has overhead: the client prepares it, the network transfers it, and the broker parses and handles it. Batching lets records share part of that cost. As a rough model:

Work per record ≈ fixed batch overhead ÷ records per batch + record-specific work.

The record-specific work still exists. Kafka must move bytes, and applications must serialize, deserialize, and process them. Batching reduces repeated overhead; it does not eliminate the payload or business logic.

In the Apache Kafka Java producer, batches form per producer and partition. A busy topic can still receive small batches if many producers spread its traffic across many partitions. Increasing a batch-size limit cannot fill a batch when too few records arrive together.

Allowing more time to accumulate a batch can add latency at light load. Under heavy load, fuller batches can reduce request pressure and queueing enough to improve observed latency. The effect depends on which cost dominates. This is why you cannot treat batching as a fixed exchange of “more throughput for worse latency.”

Compression changes another part of the equation. It spends CPU to reduce the bytes Kafka transfers and stores. Repetitive events may compress well; already compressed payloads may not. If the network is the bottleneck and producers have spare CPU, compression may help. If producer CPU is already saturated, the same choice may reduce throughput.

Concurrency matters too. An application that waits for every record’s acknowledgment before submitting another leaves little opportunity to overlap requests. Allowing bounded outstanding work can keep the pipeline busy, but the application must still observe completions and handle errors. An ever-growing list of pending sends is another queue, not evidence of higher completed throughput.

Partitions provide independent logs and units of assignment to regular consumer groups. They allow work to spread, but they do not guarantee that it spreads evenly.

Suppose orders.placed has six partitions and six reporting consumers, with each consumer handling one partition. For this example, each consumer processes records sequentially and can finish 2,000 events per second, provided the database has capacity. When the producer distributes 8,000 events per second evenly, each handles about 1,333.

Now suppose partition 0 receives half the traffic because the routing concentrates a large source there. It receives 4,000 events per second, while the other five receive 800 each.

The diagram groups the five lightly loaded partitions to keep the imbalance visible.

Partition 0 accumulates 2,000 unfinished events per second even though the group’s nominal combined capacity is 12,000. The unused capacity of the other consumers cannot automatically process its records. Adding a seventh group member does not give partition 0 a second assigned reader.

If one key dominates traffic and must remain ordered, adding partitions alone cannot divide that key’s work. Changing the key or processing records concurrently requires an application design that preserves the required ordering and tracks completed work safely. Adding partitions also does not redistribute records that existing partitions already store.

Distribution matters at the broker level as well. Several busy leaders on one broker can exhaust its network or CPU while other brokers remain lightly loaded. Cluster averages hide that imbalance. Inspect load by broker, partition, and consumer before concluding that the whole cluster needs more capacity.

More partitions can enable more parallel work, but also mean more replica state, files, and coordination. They may spread producer traffic into smaller batches. The useful question is which work can actually run independently and whether the current partition layout lets it do so.

One logical event causes more work than one write. Followers copy it, each replica stores it, and independent consumer groups read it.

Suppose the topic receives 12,000 events per second with an average serialized size of 1,000 bytes, including key, value, and headers. That is 12 MB/s before compression, using decimal megabytes. Assume measured compression reduces the record-batch stream to approximately 6 MB/s.

With replication factor three and two groups each reading the entire stream once, the simplified data flow is:

The leader replicas receive approximately 6 MB/s from producers and send 24 MB/s across replication and consumer responses. Across all three copies, the topic adds approximately 18 MB/s to replica logs before cleanup. These are cluster-wide payload estimates, not per-broker numbers or exact physical-device traffic.

The estimate excludes request headers, acknowledgments, indexes, retries, and other overhead. It assumes batches retain the same compressed representation and both groups keep up without rereading. Its purpose is to show why producer ingress alone understates demand. A third independent group adds another stream of reads without adding another stored topic replica.

The cost of those reads depends on where the data resides. The operating system’s page cache holds recently accessed file data in memory. A caught-up consumer may read cached data; an old replay may require storage-device reads and displace useful cached pages. Consequently, replaying 6 MB/s can cost more storage work than reading 6 MB/s of fresh records.

Memory, storage bandwidth, network bandwidth, and CPU interact. Giving a broker more heap within the same memory limit leaves less room for other uses, including page cache. Sequential appends are efficient, but sustained writes still have to reach storage. A short test may end before writeback pressure appears.

Replication also affects waiting. With acks=all, the producer’s confirmation depends on the in-sync replicas, so follower delays can affect acknowledgment latency. This is not a requirement to physically flush every record to every replica’s device before acknowledging it. Nor does it confirm consumer processing.

Reducing the acknowledgment requirement changes what success means. Compare performance while keeping the required durability, security, and processing behavior intact.

Return to the reporting dashboard. In a representative test, orders.placed accepts 12,000 events per second across six evenly loaded partitions. Producer acknowledgment latency remains stable, but reporting finishes only 10,000 events per second. Unfinished reporting work grows at the expected 2,000 events per second.

The stable acknowledgment latency suggests that publishing is not where the new delay appears. It does not prove that every broker resource is healthy. The next step is to compare consumer fetch timing, application queue depth, and database completion timing.

Suppose fetch responses remain fast, while waits for database connections rise. Database measurements show that write capacity is already exhausted. Increasing consumer fetch sizes would only move more waiting records into application memory. Adding brokers would not increase database completion capacity either.

The useful change must address the measured limit. In this example, the team tests batching compatible database updates while preserving its idempotency and commit rules. If database time falls and completed events per second rise, that supports the diagnosis. If the database remains saturated, adding more concurrent callers may just increase waiting.

This reasoning gives a practical sequence:

A quota can delay requests even when hardware has spare capacity. An open transaction can delay record visibility for a read_committed consumer. These are examples of configured or correctness-related waits; high latency does not always mean exhausted hardware.

Offset commits must continue to reflect safe recovery progress. Committing ahead of unfinished database work can lower reported lag while creating a data-loss risk after a crash. After successfully processing partition 0 through offset 42, the application can commit 43 as its next restart position. A crash before that commit may repeat processing, so the database operation must tolerate that repetition.

A production pipeline needs capacity for more than its healthy steady state. A broker outage shifts leadership and serving work to surviving brokers. A recovering replica copies missing data while new writes continue. A consumer failure can pause processing during reassignment and cause work after the last committed offset to run again.

These events reduce available capacity or add demand, often at the same time. Retries can add further load when resources are already under pressure. Recovery therefore depends on the same spare capacity the pipeline uses to absorb traffic bursts.

Evaluate a workload long enough for queues, cache behavior, and storage writeback to settle. Use representative event sizes, key distribution, compression, consumer processing, and security settings. Then check whether acceptable throughput and latency hold during a relevant disruption, such as a broker restart or a controlled replay.

The important result is a sustained completion rate with bounded waiting, acceptable failures, and enough headroom to recover. A brief peak that leaves a growing backlog does not establish that the pipeline can support that rate in production.

Kafka performance comes from the work and waiting along the entire event path. Define completion first, then measure both throughput and latency at that boundary. Acknowledging a write, fetching a record, and completing a business action describe different progress.

Batches reduce repeated overhead, partitions enable independent work, and replication and readers add resource demand. Uneven traffic or a slow dependency can limit one path while the rest of the system has spare capacity.

Find the growing queue and identify what limits how quickly the system processes it. Then test a change that addresses that limit. Keep correctness requirements intact and leave capacity for bursts, retries, replay, and recovery.