Choosing a partition count is a capacity decision with a lasting effect on the applications using a topic. Too few partitions can limit how quickly producers write records or consumers process them. Too many add overhead and can make an otherwise simple workload harder to operate.

There is no partition count that works for every topic. A useful choice comes from the workload, measured capacity, ordering requirements, and recovery targets. In this chapter, we’ll work through those inputs, calculate a starting estimate, and turn it into a candidate we can test.

Partitions divide a topic into separate logs. Kafka can spread their leaders across brokers, and a regular consumer group can assign different partitions to different consumers. The count therefore affects opportunities for parallel writing and reading.

For a group subscribed only to one topic, six partitions allow at most six consumers to have an assigned partition at the same time. A seventh consumer cannot share one of those assignments. Fewer consumers can read all six partitions, but each must handle the combined work of its assignments.

That is an assignment limit, not a throughput promise. Six partitions on an overloaded broker do not create additional CPU or network capacity. Six consumers writing to a saturated database cannot make that database accept more writes merely because Kafka divides their input differently.

Partition count is also separate from replication factor. Twelve partitions with replication factor three mean twelve distinct logs and thirty-six partition replicas. The extra copies support durability and recovery; they do not provide thirty-six independent reading assignments within one regular group.

The sizing examples here use ordinary topics and regular consumer groups. They assume each consumer processes its assigned records sequentially. Applications with internal worker pools or Kafka share groups have different processing arrangements and need measurements that reflect those arrangements.

Begin with the traffic the system must handle, including peaks that last long enough to matter. An average taken across an entire day can hide a busy hour during which the topic continuously accumulates backlog.

Measure both records per second and serialized bytes per second. Record count helps estimate application processing work, while byte volume helps explain broker storage and network demand. Keep the byte definition consistent: uncompressed key-and-value bytes, compressed batches, and total network traffic are different measurements.

For each consumer group, measure completed business processing, not just how quickly the client polls records. Include deserialization, validation, database calls, retries, and whatever must finish before the application can safely advance its checkpoint.

Measurements should resemble the intended deployment. Record sizes, keys, compression, acknowledgment settings, replication, encryption, broker hardware, and downstream services can all affect the result. A local single-broker plaintext experiment cannot establish production capacity for a replicated cluster using encrypted connections.

Choose a sustainable design rate below the point where latency or errors become unacceptable. That leaves headroom, spare capacity for variation. The appropriate margin depends on the workload and service objectives; a fixed percentage is not a Kafka rule.

Alongside traffic and processing rates, record the expected outage duration, the catch-up deadline and how much traffic growth to plan for. These requirements often change the answer more than a small improvement in producer throughput.

Consider an orders.events topic keyed by order ID. The following values are hypothetical load-test results that illustrate the calculation. We did not run these tests for the chapter, and the values are not Kafka benchmarks.

Here, one MB means 1,000,000 bytes. The write-rate test and the input calculation use the same serialized-byte basis. The test includes the intended replication, acknowledgment, and security settings, while measuring application input rather than multiplying it by replica traffic.

Assume keys distribute traffic reasonably evenly and each consumer can reach its design rate on one partition. Also assume enough broker and downstream capacity exists to sustain several such streams simultaneously. These assumptions need validation at the candidate partition count.

The write-side estimate is:

For a sequential consumer group, estimate the number of consumers needed from its completed processing rate, then provide at least that many partitions if each consumer is to have an assignment:

The diagram shows why we compare the group requirements rather than add them. Both groups independently read the same topic; they do not divide one pool of partitions between them.

Six is a starting lower bound under these assumptions, not a completed design. The two groups still add their read traffic and processing demands to the system. Taking the maximum for partition assignments does not mean their resource use is free.

Assignment granularity matters as well. With six evenly loaded partitions, each receives 2,000 records per second. With four notification consumers, some consumers would receive two partitions, requiring 4,000 records per second from those consumers—above their 3,000-record design rate. Six notification consumers are a more suitable steady-state arrangement for this particular layout unless further measurements support another arrangement.

Dividing aggregate traffic by per-consumer capacity estimates required capacity. Checking actual partition assignments tells you whether the workload can use that capacity.

A group that can process exactly as fast as records arrive has no spare capacity to clear a backlog. After an outage, it can remain behind indefinitely even though it no longer falls further behind.

Suppose reporting may be offline for ten minutes while the topic continues receiving 12,000 records per second. The recovery requirement is to clear that backlog within twenty minutes after processing resumes, while new traffic continues at the same rate.

The diagram separates handling new arrivals from clearing old work. Both consume reporting capacity during recovery.

Recovery raises the reporting estimate from six to nine partitions and consumers. It also requires the reporting database to accept 18,000 records’ worth of work per second. Additional partitions cannot satisfy the recovery target if that downstream limit remains 12,000.

For this example, twelve partitions are a reasonable candidate to test. With twelve reporting consumers sustaining the assumed rate, the nominal capacity is 24,000 records per second. At 12,000 ongoing arrivals, that leaves 12,000 records per second to clear the backlog, which would drain 7.2 million records in ten minutes under the simplified model.

This gives room beyond the twenty-minute recovery requirement and allows some growth. It does not prove the target will hold after a consumer or broker failure. Lost capacity, uneven assignments, rebalances, and temporary errors can change the result. Test the candidate with the actual number of running consumers, including any scaling delay.

At the original steady input rate, twelve evenly loaded partitions receive 1,000 records per second each. Four notification consumers with three partitions apiece would each handle 3,000 records per second, matching their design rate. Reporting can use twelve consumers to retain its planned recovery capacity. The groups do not need the same number of instances.

Retention must cover the outage and the age of the backlog during recovery, with a suitable margin. A group cannot catch up on records that cleanup has already removed.

All of the calculations so far assume a reasonably even workload. Real topics can be uneven in record volume, byte volume, or processing cost.

Suppose the same platform instead keys all order events by tenant ID. One large tenant generates 8,000 of the topic’s 12,000 records per second. With stable key-based routing, that tenant’s records all reach one partition.

The diagram groups the remaining partitions for readability. Their spare capacity cannot divide the busy tenant’s single-partition stream between regular group members.

Even twelve partitions and twelve consumers cannot solve this under the assumed sequential processing model. The busy partition receives work faster than its consumer can complete it.

At the illustrative record size, that tenant also contributes 8 MB per second to one partition, exceeding the 4 MB write design rate. Key skew can invalidate both sides of the capacity estimate.

More distinct keys do not necessarily mean balanced traffic. Inspect the busiest keys and partitions, and include expensive event types in the load test. A partition containing fewer records can still be the slowest if each record requires much more work.

If order-level sequencing is sufficient, using order ID rather than tenant ID may allow the large tenant’s unrelated orders to spread out. If tenant-wide sequencing is essential, splitting that tenant across keys would weaken a requirement. Improving processing capacity or changing the application design may then be necessary.

Do not use a large partition count to hide an unresolved key-design constraint. A single key remains a single routing unit under the ordinary keyed strategy.

Extra partitions are useful when they provide capacity or flexibility the application can use. They also add work throughout the system.

Every partition replica has log files, indexes, and operational state. More partitions mean more metadata for the KRaft controllers and brokers to manage, and more logs involved in replication, recovery, and maintenance. Consumer groups also have more assignments and per-partition positions to track.

For the twelve-partition candidate with replication factor three, Kafka maintains thirty-six replicas. On six brokers, an evenly balanced placement would average six replicas per broker for this topic. The cluster must also account for every other topic and internal workload; sizing one topic in isolation is insufficient.

Partitioning the same incoming data more finely does not multiply its payload volume. It does increase per-partition overhead. Replication, retention, and the event volume remain major inputs to storage capacity.

Producer batching can become less efficient too. A batch groups records for a partition so the producer can send them efficiently. Spreading modest traffic over many partitions can produce more small batches, especially when latency requirements limit how long producers can wait. The effect depends on producer count, routing, record size, compression, and client settings.

There is also a processing trade-off. More partitions can divide state and assignments into smaller units, but managing and restoring many units has a cost. That makes “create hundreds now, just in case” a decision to justify with expected needs and measurements.

A broker count does not dictate the topic’s partition count. Multiples of the broker count can make some layouts easier to balance, but they are not a Kafka requirement and do not prove the load is balanced. The worked capacity and recovery targets justify testing twelve partitions; the fact that twelve is a multiple of the broker count does not.

Leave room for a realistic growth horizon rather than attempting to predict the topic’s entire lifetime. For example, twelve reporting consumers at the illustrative design rate would have only 8,000 records per second of spare capacity if arrivals grew to 16,000. Growth consumes recovery headroom as well as steady-state headroom.

Kafka supports adding partitions, but existing records stay in their original partitions. New partitions start empty. Expansion can create more destinations for future traffic, yet it does not redistribute a backlog that is already concentrated in an old partition.

Increasing the count can also change key-to-partition mappings. Producers discover the new layout through metadata updates, potentially at different times, while consumers must discover and receive assignments for the new partitions. Applications requiring per-key order or maintaining partition-local state need a coordinated transition.

Starting positions matter. New partitions have no previous group checkpoint. A consumer using auto.offset.reset=latest can skip records producers write to those partitions before it discovers them and chooses its initial position. Decide explicitly how consumers should begin reading new partitions.

Kafka does not support reducing a topic’s partition count in place. If a smaller count becomes necessary, moving to a new topic requires planning how to move the data, update applications, and switch traffic to the new topic. This makes unnecessary over-allocation harder to reverse than adding idle consumer instances.

Adding brokers is a separate capacity change. Existing replicas need placement on those brokers before the new machines can help carry their load. Expanding partitions, moving replicas, and adding consumers should each address an identified limit rather than serve as interchangeable scaling actions.

Treat twelve partitions as the input to a realistic test, not the conclusion of the arithmetic. A useful validation establishes whether the complete system meets its targets with that layout.

Run the expected sustained peak with representative key distribution and record sizes. Include both consumer groups so broker reads, replication, and downstream work compete for resources as they would in normal operation. Check the busiest partition and consumer rather than relying only on averages.

Then build a controlled backlog in a test environment and resume processing while writes continue. Verify that reporting clears it within the target time, commits progress safely, and produces correct results. A consumer that reports low lag by committing before its work finishes has not demonstrated recovery capacity.

Exercise the failures the design must tolerate. A consumer loss can change assignment balance; a broker loss can move leaders and concentrate traffic on surviving machines. Measure interruption time and catch-up afterward rather than assuming the healthy-state rate still applies.

If the system misses the target, locate the constraint. More partitions may help when available consumers lack enough independent assignments. They will not fix a hot key, a slow database, or a cluster already saturated in aggregate. Repeat the relevant test after changing the constraint, and compare nearby candidate counts if the benefit of extra partitions is uncertain.

Record the chosen count together with the measured rates, peak assumptions, recovery target, key distribution, consumer deployment, and conditions that trigger a review. That explanation is more useful than a partition number without context.

Estimate partition needs from measured write capacity and each consumer group’s completed processing rate, then include backlog recovery and realistic growth. Check actual assignments and key distribution: aggregate capacity is useful only when the workload can reach it.

Extra partitions add operational cost and can affect batching, while expansion changes future routing without moving existing records. Choose a count that meets a tested workload and recovery target with justified headroom, and revisit it when those assumptions change.