Producer Batching

A producer that sends every event in a separate request spends much of its effort repeating the same work: building requests, transmitting headers, and handling responses. Collecting several records before sending them lets that work serve more events.

The trade-off is waiting. A record may spend a little longer in the producer so that other records can join it. Whether that helps depends on the arrival rate, partition distribution, record sizes, and how much latency the application can tolerate.

In this chapter, we’ll examine batching in the Apache Kafka Java client 4.3.1. We’ll follow order events into batches, explain the main settings, and look at how to evaluate a change without mistaking a larger buffer or a longer delay for better performance.