Capacity Planning and Partition Sizing

A Kafka cluster can have enough disk space for its retained records and still struggle during a traffic peak. It can handle that peak while every broker is healthy, then fall behind when one broker fails. A topic can also have plenty of partitions while one consumer group lacks the capacity to process them.

Capacity planning connects these constraints. The goal is to choose a cluster and partition layout that can store, serve, process, and recover the expected workload within its requirements.

In this chapter, we’ll work through a sizing example from traffic assumptions to a candidate deployment. We’ll calculate storage and network demand, derive partition needs from measured rates, and check whether the layout leaves room for recovery.