An order topic may start with a small stream of events and grow to serve several busy applications. As traffic increases, Kafka needs a way to spread storage, incoming writes, and consumer work across more resources.

Partitions provide that division. Each partition holds part of a topic’s records in its own ordered log. In this chapter, we’ll look at how those logs work, where they live, and how they let applications process a topic in parallel. We’ll also examine the limits of that parallelism and inspect a topic’s partition layout.

A partition is an ordered, append-only log within a topic. Append-only means producers add new records at the end rather than editing records Kafka already stores there. Kafka’s cleanup policies can still remove older data.

A topic has one or more partitions with identifiers starting at 0. A topic with three partitions has partitions 0, 1, and 2. Those numbers identify the logs; they do not indicate priority or processing order.

Suppose orders.placed has three partitions. Each record contains an OrderPlaced event and uses the order ID as its key. Here is a small portion of their stored history. The diagram omits earlier records, and the partition choices are illustrative rather than the result of a routing calculation shown here.

Each arrow shows increasing offset order within one partition. There are no arrows between partitions because Kafka does not define a single log order across them.

An offset identifies a record’s position within its partition. Offset 42 in partition 0 says nothing about which record Kafka appended first in partition 1. Another partition can have its own offset 42, describing a different record.

Each partition grows independently. One can receive thousands of new records while another receives none. Their offsets do not need to advance together, and the partitions do not have to contain equal amounts of data.

To identify a stored record, include the topic, partition, and offset. Within this cluster and topic’s lifetime, orders.placed, partition 0, offset 42 identifies the record for ord-1042 shown above.

For each record it sends, a producer selects one partition of the destination topic. The application can specify the partition directly, or the producer can choose it using its configured routing behavior, often based on the record key.

Kafka appends the record to the selected partition. Kafka does not split its JSON value across several partitions or place a separate business record in every partition. Replica copies of the selected partition are a different matter: they preserve copies of the same log.

Several producer instances can write to the same partition. For example, two order-service instances may both send events to partition 0. The broker leading that partition places the accepted records into its log order. Several producer connections can share a partition.

The producer also does not choose a partition based on which consumer is currently free. Producers write to the topic’s partitions, while consumer groups assign readers to those partitions separately. Starting another reporting consumer does not redirect the producer’s next event to it.

This separation lets writing continue while an application is offline, provided Kafka remains available and has capacity. The application can catch up later while Kafka still retains the records it needs.

Partitions let Kafka spread a topic across brokers, the servers that store and serve its records. Each partition has a leader replica that accepts writes and may have follower replicas that copy the log. A replica is one copy of a partition, and the replication factor counts all copies, including the leader.

Consider a three-broker layout for our topic with replication factor 2. Each partition has one leader and one follower. The arrows show replication data moving from a leader to its follower; followers obtain that data by fetching it.

This topic has three distinct partition logs and six replicas in total. Partition 0 on Broker 2 copies partition 0 on Broker 1; it does not hold an extra portion of the topic’s records. The layout illustrates the roles, not a production replication recommendation.

Because the three leaders are on different brokers, writes to different partitions can use different machines. Consumers normally fetch from those leaders too, although Kafka also supports configured follower fetching.

A topic does not need one broker per partition. A broker can lead many partitions, and a topic can have more partitions than there are brokers. Conversely, adding brokers does not turn one partition into several independently writable logs. Its leader still handles that partition’s writes.

The three counts describe different resources: partitions divide the data, replicas provide copies, and brokers provide the machines that host those copies. Increasing one does not automatically increase the others.

A consumer group lets instances of an application share the work of reading subscribed topics. For regular consumer groups, once assignments settle, each partition has one assigned consumer within the group. One consumer can receive several partition assignments.

Suppose order-reporting contains two consumers and subscribes only to our three-partition topic. One possible assignment gives partitions 0 and 1 to Consumer A and partition 2 to Consumer B. A separate notifications group can read all three partitions independently.

The arrows below represent reading assignments, not copies of the stored data. Both groups fetch from the same partition logs.

Reading partition 0 in reporting does not prevent notifications from reading it. The one-reader rule applies within each regular group, not across the entire cluster. Kafka share groups use a different model; the assignments here describe regular consumer groups.

If a third reporting consumer joins, an assignment can give each consumer one partition. If a fourth joins, at least one reporting consumer has no partition from this topic. Three partitions provide at most three simultaneous assigned readers in that group for this topic.

That limit describes Kafka’s reading assignments, not every thread an application may run. A consumer can hand records to worker threads, but then the application must preserve any required processing order and track completion safely before committing progress.

Nor does equal partition ownership guarantee equal work. A consumer that reads one busy partition may do more processing than another that reads two quiet ones. Record volume, record size, and the cost of handling each event all affect the workload.

Partitions make parallel work possible by giving each log its own order. A consumer reading our partition 0 forward encounters offset 42 before offset 43. Kafka provides no corresponding rule that requires a consumer to process partition 0 offset 42 before partition 1 offset 17.

For reporting that adds totals from unrelated orders, that independence may be acceptable. If an application must apply several changes to one order in sequence, those records need consistent placement in a partition, suitable producer behavior, and processing that preserves the required order. Merely putting all records in the same topic is insufficient.

Even a single consumer reading all three partitions does not create a guaranteed order across them. It combines data from separate logs as it fetches and processes it. The order it happens to print is not a topic-wide ordering guarantee.

Kafka also tracks consumer progress per partition. Assume the six records in our first diagram are the latest records, with no further writes, and reporting saves progress only after completing all earlier work in each partition. It could have the following committed positions:

These are three independent checkpoints, not one “topic offset.” Reading or committing past a record does not remove it from the partition. Cleanup controls how long it remains available.

Suppose Consumer A fails. The reporting group can reassign its partitions through a rebalance, a change in which consumers own which partitions. A replacement can resume partition 0 at 43 and partition 1 at 19, provided those saved positions remain valid and Kafka retains the needed history.

No partition data needs to move between brokers just because a different consumer takes over. However, if Consumer A updated a report before failing to commit its progress, its replacement may repeat that update. Partition assignment and offset tracking do not make external actions happen exactly once.

Partitions provide opportunities for parallel work, but they do not guarantee that the workload will use those opportunities evenly. If most records land in partition 0, that partition can become a hot partition: it carries much more traffic or processing work than the others.

Adding a fourth consumer to our reporting group cannot split ownership of partition 0 between two group members. Adding empty partitions also does not move its existing backlog elsewhere. The bottleneck depends on where new records go and how quickly the application can process the busy log.

Changes to brokers, partitions, and replicas therefore solve different problems:

Kafka supports increasing a topic’s partition count, but it does not support reducing that count in place. New partitions start without the topic’s existing records. Producers and consumers discover the changed layout through metadata updates, and subscribed groups may need to adjust assignments.

Increasing the count can also change where key-based routing sends future records. Older records remain in their original partitions, so records for the same key can end up spanning the old and new layout. Treat expansion as a planned change, especially when processing depends on keeping related records in order.

Broker failure is a separate concern. In our replicated layout, if Broker 1 fails, partition 0 loses its leader. If the controller quorum is available and Broker 2’s replica is eligible, Kafka can elect it as the new leader. Clients refresh their metadata and continue using partition 0 through its new leader. A leadership change does not change the partition’s identity.

Without an available eligible replica, that partition may remain unavailable. Other partitions with available leaders may continue serving requests, though shared resources and application behavior can spread the impact. Partitioning alone provides no backup copies; recovery and durability depend on replication, configuration, and the failure itself.

To see partition metadata directly, use Kafka’s topic tool. These commands assume a running container named kafka-local with Kafka 4.3 tools and a single combined broker and KRaft controller. The broker must be reachable at localhost:9092 inside the container.

Run the commands from a Bash-compatible terminal on your host. The tools run inside the container, so you do not need to install Kafka on the host. This local setup uses plaintext connections and replication factor one; it has no backup replica if the broker loses its data.

Create a fresh topic for inspection:

If that topic already exists, choose another name and use it in both commands. Then describe it:

The summary should report PartitionCount: 3 and ReplicationFactor: 1. The partition entries should identify partitions 0, 1, and 2.

For example, if the local broker ID is 1, these selected fields would look like this. This is a simplified excerpt; the actual output includes the topic name and may contain additional fields.

Leader identifies the broker currently leading the partition. Replicas lists the IDs of brokers that hold copies of it. Isr lists the in-sync replicas, which Kafka considers sufficiently caught up with the leader. In this single-broker example, each list contains only the leader.

The repeated 1 is a broker ID, not an offset or a count of stored records. Your broker may have a different ID. All three leaders being on the same broker is expected here: the topic has three logs, but the cluster has only one machine hosting them.

Creating the topic does not create consumers or sample events. This command verifies the partition layout, not parallel application processing. Leave the topic intact when finished; no partition-count change is needed to inspect it.

A topic’s partitions are separate ordered logs. Kafka appends each record to one partition, and its offset identifies a position within that log. Kafka can spread partition leaders across brokers and assign different partitions to different consumers in a regular group.

Parallelism and ordering meet at the partition boundary: logs can advance independently, while each keeps its own order and consumer progress. Partitions divide data, replicas copy it, and brokers host it. Understanding those roles helps explain why more consumers or machines do not automatically fix a busy partition or provide stronger durability.