A Kafka broker can accept new order events, serve older records to a reporting application, and copy data from another broker at the same time. Each activity uses the broker’s network connections, memory, CPU, and storage. When one becomes slow, the effects can reach applications doing quite different work.

Understanding the broker’s internal structure helps explain where that time goes. We’ll look at how requests enter a broker, how its components handle them, and how a record becomes available to consumers.

The implementation details here use Apache Kafka 4.3.1 with KRaft. The examples assume local broker storage, non-transactional records, and ordinary consumers fetching from partition leaders. Kafka also supports configurations such as follower fetching and tiered storage, which change parts of the read path.

A broker is a Kafka server process that stores partition replicas and handles Kafka protocol requests. It usually hosts replicas for many partitions across several topics. Some are leaders that accept producer writes; others are followers that fetch records from leaders on other brokers.

Suppose Broker 1 leads partition 0 of orders.placed and follows partition 1. A write to partition 0 goes through Broker 1’s leader replica. Meanwhile, Broker 1 fetches partition 1 records from that partition’s leader and appends them to its local follower replica.

Both logs occupy storage on Broker 1, but they have different responsibilities. Leadership belongs to a partition, so the same broker can lead one partition and follow another.

The broker also handles metadata requests and may act as a coordinator for consumer groups or transactions. These responsibilities live in the same process and share its resources.

Brokers and Controllers

KRaft controllers manage cluster metadata, including broker registrations, replica assignments, and partition leadership. Brokers maintain a local view of that metadata and use it to serve requests and manage their replicas.

This separates two kinds of work. Brokers handle the data plane, the traffic that writes, reads, and replicates topic records. Controllers manage the control plane, the decisions that govern the cluster’s structure.

A Kafka server can run the broker role, the controller role, or both. A combined process is convenient for local learning. Separate processes let operators give controllers independent resources and maintenance schedules.

The controller’s metadata log and a topic’s partition log contain different information. Updating the leader assignment for orders.placed-0 is a metadata operation. Appending an order event is a broker operation on that partition’s data log.

Several components cooperate to handle a request. The network layer manages connections. Request handlers interpret Kafka operations and call the relevant services. Replica and log management provide access to the partition data the broker stores.

The diagram shows these responsibilities inside one broker. It is a component view, so the arrows represent interactions rather than a strict execution sequence.

Replica management tracks the local replicas and their leader or follower state. It connects requests to the appropriate partition, manages replication activity, and tracks when replicated data can become visible to consumers.

Log management handles the local logs and their lifecycle. Each replica’s log consists of files, including record data and indexes that help Kafka locate records. Background work manages tasks such as retention cleanup and, for compacted topics, log cleaning.

Coordinators also need durable state. They persist it through internal Kafka topics, whose partitions use the same underlying storage and replication mechanisms as application topics.

These components are parts of the broker software. You don’t deploy a separate replica manager or log manager service for each topic.

Before the broker can append or fetch data, it has to receive a request. Kafka exposes listeners, named network endpoints with associated security settings. A deployment can use different listeners for application traffic and communication between brokers.

The listeners property specifies where the server binds. The advertised.listeners property specifies the broker addresses clients should use after discovery. Those addresses must be reachable from the intended client network.

For example, an application might reach a bootstrap broker successfully, then discover a partition leader whose advertised internal hostname the application cannot resolve. The initial connection works, but requests to that leader fail. Bootstrap connectivity alone does not establish access to the brokers that will serve the application.

From Socket to Handler

The broker separates network work from request processing. An acceptor handles new connections for a listener and assigns them to network threads. Each network thread manages multiple connections, receives requests, and sends responses.

Completed requests enter a request queue. A pool of request handler threads takes work from that queue and dispatches it to the appropriate broker component. Kafka also calls these handlers I/O threads, though their work can include validation and coordination as well as storage access.

This diagram follows a request on an established connection. Connection acceptance happens before the path shown here.

The queues let the network and handler threads make progress independently. The broker does not allocate a dedicated request handler to every client connection or partition.

The settings num.network.threads and num.io.threads control the relevant thread pools. Increasing them can help when those pools limit throughput and the machine has spare capacity. It cannot make a saturated disk or network interface faster.

The request queue is bounded. If requests arrive faster than handlers can process them, queueing time increases and the broker eventually limits further intake. Clients can then experience longer response times or timeouts. A larger queue allows more waiting work; it does not increase the rate at which the broker finishes that work.

Delayed Operations

Some requests cannot finish immediately. A write with acks=all may need replication to advance. A consumer fetch may need more data to become available.

Kafka can register a delayed operation, keeping the information it needs to finish the request when it meets its completion condition or its timeout expires. Kafka calls the structures tracking these operations purgatories.

This allows a handler to continue processing other requests while the operation waits. A delayed fetch therefore does not require a handler thread to sit idle for the entire wait. Request handling can still block during work such as storage access; delayed operations address specific waits, not every source of latency.

Suppose an order service publishes an event to orders.placed, partition 0, with key ord-1042, event ID evt-7001, and a source header containing order-service. The key, value, and header values arrive as bytes inside a record batch.

Broker 1 leads the partition. Brokers 2 and 3 hold follower replicas. Assume all three remain in sync, the producer uses acks=all, and the topic uses min.insync.replicas=2.

The producer has already chosen the partition and found its leader. Broker 1 receives a Produce request naming that partition; it does not choose a partition by hashing the record key again.

Validation and Append

The broker checks whether it allows the operation and whether it can accept the data for the requested partition. Relevant checks include authorization when configured, partition leadership, record-batch validity, and size limits. It validates Kafka’s record structure without interpreting whether the order’s business contents are correct.

For a valid new batch, the leader appends the records to its local partition log and assigns offsets. Suppose our event receives offset 42. The next offset in that log is now 43.

An append writes through the filesystem and normally benefits from the operating system’s page cache, memory the operating system uses to cache file contents. Kafka does not generally force a physical disk flush for every acknowledged record. Replication and the configured acknowledgment policy are central to its durability model; acks=all does not mean every replica has performed an fsync for that record.

Replication and Response

Followers continuously fetch from the leader. When Brokers 2 and 3 receive the record, they append it with the same offset, 42, in their copies of partition 0. The leader learns about follower progress through replication fetch requests.

In-sync replicas, or the ISR, are replicas Kafka considers sufficiently caught up with the leader. Under our assumption that all three stay in the ISR, a successful acks=all response waits for all three to have the write. The minimum of 2 is an admission requirement for this acknowledgment mode, not a rule to acknowledge as soon as any two of the three replicas have it.

The sequence below separates appending from waiting for replication.

The response describes the result of the write to Kafka. The reporting application may not have fetched the event yet.

If the producer never receives the response, the producer can time out even though the write succeeded. Kafka does not roll back an append because the client missed its acknowledgment. Producer idempotence can prevent duplicates from the producer’s own retries. Before sending the event again, the application must decide how to prevent duplicate business effects.

A consumer fetch identifies a partition and the offset from which to read. Suppose the reporting consumer requests partition 0 starting at offset 42.

The request follows the network and handler path to replica management. The broker checks the request and reads eligible records from the local log, subject to the fetch limits. Recently written data may already be in the page cache. Reading older data that is absent from the cache requires storage I/O.

Read Visibility

A record can exist in the leader’s log before it is available to ordinary consumers. The high watermark marks the replication boundary: records with offsets below it are replication-committed, and ordinary consumers can read them subject to transaction isolation.

In our example, if the leader has appended offset 42 but the high watermark is still 42, that record is not yet eligible. Once replication advances the high watermark to 43, the broker can return the non-transactional record at offset 42.

Replication-committed here describes the partition’s data. A consumer’s committed offset describes its saved processing progress. These are separate states, even though both use the word “committed.”

A consumer using read_committed also respects transaction visibility, which can hold its readable boundary behind the high watermark. Our non-transactional example avoids that additional condition.

Waiting for Data

When too little eligible data is available, the broker can delay the fetch until enough data arrives or the requested wait expires. The consumer settings fetch.min.bytes and fetch.max.wait.ms influence this behavior.

This is a long poll: the consumer has already requested data, and the broker waits before answering. It reduces repeated empty requests while letting the broker return batches of records.

The sequence below shows the read and the two ways the fetch can complete.

Returning offset 42 does not delete the record or automatically commit offset 43 for the consumer. The application processes the event and manages its progress separately. Other consumer groups can fetch the same stored record at their own pace while Kafka retains it.

Serving application records is only part of a broker’s workload. Its coordinator responsibilities and background tasks continue alongside Produce and Fetch requests.

A group coordinator handles consumer group operations and committed offsets. Kafka stores group state and offset commits in __consumer_offsets. The broker responsible for a group can differ from the broker serving that group’s application records.

For example, Broker 1 might serve orders.placed-0 while Broker 3 coordinates order-reporting. Fetching order events uses Broker 1; group coordination and offset commits use Broker 3. A coordinator problem can interrupt group management even while Broker 1 remains able to serve the partition.

Brokers can also host transaction coordinators, which manage Kafka transaction state that Kafka stores in __transaction_state. This is separate from the KRaft controller’s responsibility for cluster metadata.

Meanwhile, follower fetchers copy records from other brokers, log maintenance reclaims eligible storage, and the broker applies metadata changes. For example, a leadership change can cause a local replica to stop accepting producer writes and begin following a new leader.

These activities have different purposes but compete for the same machine’s resources. A broker with few producer writes can still be busy copying follower data, serving historical reads, or cleaning compacted logs.

The broker’s architecture gives you a useful way to reason about latency: identify where work is waiting and what resource would let it move forward.

If requests spend time in the request queue, handler capacity or the work those handlers perform deserves attention. If local appends are quick but acks=all writes remain slow, replication progress may be delaying completion. If historical fetches slow down while recent reads remain fast, storage reads and page-cache behavior are worth investigating.

These are starting points for diagnosis, not conclusions from one symptom. Network congestion, CPU pressure, quotas, and storage delays can affect several stages at once.

Consider a reporting application replaying weeks of order history. Its reads may require substantial disk I/O and consume outgoing bandwidth. The same broker may need those resources to replicate fresh orders. A replay can therefore increase write latency for another application, even though the applications use different consumer groups.

Partitions provide separate ordered logs, but replicas that share a broker share resources. Adding brokers creates capacity only when partition placement and traffic allow the work to use it. A heavily used partition can still concentrate work on its leader.

This is why useful broker analysis includes leader traffic, follower traffic, coordinator work, and background storage activity. Counting producer requests alone leaves much of the broker’s workload unexplained.

A broker combines network handling, request processing, partition storage, replication, and coordination in one process. KRaft controllers manage cluster metadata, which brokers use to carry out their local responsibilities.

Produce requests pass through validation and local append before completing under the requested acknowledgment policy. Fetch requests return records within the applicable visibility boundary. Delayed operations allow replication and data-availability waits without occupying a handler for the whole wait.

All of this work shares the broker’s resources. Understanding the request path helps explain both what Kafka has confirmed and where a slow operation may be waiting.