A reporting service reads order events from Kafka and writes them to a database. Its main loop may look simple: get some records, process them, and repeat. Behind that loop, the consumer discovers brokers, fetches batches, converts bytes into application values, and keeps track of where it has reached.

Understanding those steps explains why a consumer can read ahead of its application, why a restart can repeat completed work, and why a healthy broker connection does not prove that processing is making progress.

We’ll follow records from orders.placed into a reporting application using the Apache Kafka Java client 4.3.1. The example uses an ordinary consumer group, non-transactional records, and explicit offset commits. Where threading matters, we’ll describe group.protocol=classic and briefly distinguish the newer consumer protocol.

The consumer is a client library running inside your application process. Brokers store and serve the records; the consumer manages the work needed to read them.

The application calls poll() to obtain records. To support that call, the client maintains cluster metadata, partition assignments, reading positions, network connections, and buffers for fetched data. It also communicates with Kafka to maintain group membership and save offsets.

The diagram shows the main responsibilities. These are logical components, not a promise that each box has its own thread or maps to one Java class.

The boundary between poll() and application processing matters. Once poll() returns, the application decides what to do with the records. The client cannot tell whether a database transaction succeeded or whether the application calculated an order total correctly.

Kafka also keeps the stored records independently of this consumer. Reading an event does not remove it. Another application can read the same retained event using its own consumer group and maintain separate progress.

Before the consumer can fetch anything, it needs to know which partitions to read and which brokers serve them.

Discovering the Cluster

The consumer starts with bootstrap.servers, a list of addresses it can use to discover the cluster. It obtains metadata describing topics, partitions, and broker addresses, including each partition’s current leader.

Suppose orders.placed has three partitions. Broker 1 leads partitions 0 and 2, and Broker 2 leads partition 1. Even if the consumer bootstraps through Broker 3, it normally fetches these records directly from Brokers 1 and 2. The bootstrap broker does not forward all subsequent traffic.

This example uses the normal leader-fetching path. Kafka also supports fetching from preferred replicas with suitable configuration. That changes which replica serves a read, while the consumer still needs metadata and connections to the relevant brokers.

If leadership changes, the client refreshes its information and redirects requests. The application normally does not maintain its own partition-to-broker lookup. It does, however, need network access to the addresses Kafka advertises, beyond just the initial bootstrap address.

Joining a Consumer Group

Our reporting service subscribes to orders.placed with group.id=order-reporting. A subscription expresses which topics it wants to read. An assignment identifies the particular partitions this consumer is responsible for reading.

If the group contains two consumers, one possible assignment is partitions 0 and 2 to Consumer A and partition 1 to Consumer B. The exact result depends on the assignment policy. Once the assignment settles, each partition belongs to one consumer within this ordinary group.

A broker acting as the group coordinator manages group membership and receives offset commits. Its role is separate from serving the order records, although one broker can perform both roles.

For Consumer A, those interactions might look like this:

The coordinator does not relay these record batches. A consumer can therefore have separate connections for fetching data and coordinating its group. The KRaft controller is not a proxy for either interaction.

Applications can also use assign() to choose partitions directly. That bypasses automatic group assignment, so the application must arrange who reads each partition and how another process takes over. Our example uses subscription-based group management.

Kafka consumers pull data: they request records starting from specified offsets. A broker responds with available record batches, subject to the request’s limits and the records the consumer has permission to read.

Suppose Consumer A has an established position of 42 for partition 0. Its fetch request asks for data beginning there. A request to Broker 1 can include both partitions 0 and 2, each with its own offset. Their records remain separate partition streams even when they share a request.

Fetching batches spreads network and protocol overhead across many records. If there is little data available, the broker can wait briefly before responding. We call this long polling: the request stays open while the broker waits for enough data or for its wait limit to expire.

For ordinary reads from local broker storage, fetch.min.bytes influences how much data the broker waits for, and fetch.max.wait.ms bounds that wait when insufficient data is available. Waiting for more data can improve request efficiency at the cost of latency for a quiet topic.

Fetch Responses and Poll Results

The consumer buffers fetched data locally. A later poll() can return records from that buffer, so one call to poll() does not necessarily correspond to one new network request.

Consider a response containing 300 records across Consumer A’s assigned partitions. With max.poll.records=100, the client can return those records over several polls. The setting limits records the client returns to the application per call; it does not cap a fetch response at 100 records.

Byte-oriented settings such as fetch.max.bytes and max.partition.fetch.bytes govern response sizes. They allow an oversized first batch in defined circumstances so consumption can continue. They are not hard bounds on the consumer process’s total memory use, which also includes decompressed data, deserialized values, and application work.

This separation lets the network and application use different batch sizes. The broker can serve a useful amount of data while the application takes smaller portions for processing. A small poll result alone does not prove that the client has buffered little data in memory.

The broker stores keys and values as bytes. A deserializer converts those bytes into the types the application expects.

Suppose partition 0 contains offset 42 with key ord-1042, a source header containing the UTF-8 bytes of order-service, and this JSON value:

The amount is in paise, so the order total is INR 1,299.00. With StringDeserializer configured for both key and value, the client returns Java strings. It does not turn that JSON string into an order object or validate the currency and amount. The application must parse and validate it, or use an appropriate value deserializer.

Each returned ConsumerRecord includes the topic, partition, offset, timestamp, headers, key, and value. Keeping the partition and offset with the event makes failures easier to diagnose: “invalid amount at orders.placed, partition 0, offset 42” identifies a specific stored record.

Application and Background Work

In the Java client’s classic implementation, the thread calling poll() drives normal fetch activity and deserializes records before returning them. A background heartbeat thread supports group membership. This differs from a producer’s background sender: the classic consumer does not continuously fetch new records merely because the consumer object exists.

The implementation that group.protocol=consumer selects uses a background network thread and exchanges events with the application thread. Deserialization and application processing still affect how quickly the application can receive and handle records. The diagrams here describe responsibilities both implementations share, without assuming identical threading.

A deserialization failure can cause poll() to throw before the application receives the affected record. Fetching the same unchanged bytes again does not fix a format mismatch. The application needs a deliberate response, such as fixing its deserializer or recording and handling the invalid event through an agreed failure policy.

An empty poll result also needs careful interpretation. It can mean the consumer is waiting for an assignment, has caught up, or has not obtained data within the call’s wait. It does not prove that the topic contains no records.

A consumer needs both a position for its current read and a saved position for recovery. These can differ while the application is working.

The consumer position identifies where subsequent delivery to the application will continue. It advances as poll() returns records. The committed offset is the group’s saved restart position for a topic partition. Kafka stores these commits in its internal __consumer_offsets topic.

Application completion is a separate fact. For our reporting service, a record is complete when its required database work has succeeded. Kafka does not automatically learn that fact from the database.

Assume partition 0 contains ordinary records at consecutive offsets, and the group has committed 42. A poll returns offsets 42, 43, and 44. The consumer position becomes 45, even before the application begins its first database write.

The table follows that batch with automatic commits disabled:

Committing 45 means the group should resume there. It does not mean “record 45 has finished.” Offsets are partition-specific, so partition 2 has its own positions and commits.

The consecutive offsets make this example easy to follow. Real logs can have gaps, including those caused by compaction or transactional records. Applications should use the positions the client supplies and avoid treating an offset as a count of processed business events.

If an assigned partition has no valid committed position, the consumer needs a starting-position policy. auto.offset.reset controls that fallback, for example choosing the earliest available position, the end, or an error. It does not override a valid saved position on every restart.

The separation between processing and committing creates a failure window. Suppose the reporting application successfully writes the result for the record at offset 42 to its database, but the process crashes before saving further progress in Kafka.

This sequence shows what the replacement consumer can observe. Assume the group’s committed offset remains 42 and Kafka still retains the record:

Kafka returns the original record again at its original offset. No producer needs to republish it. The saved progress simply does not include the completed database work.

The reporting application should make this repeat safe. One approach is to store evt-7001 in a processed-events table with a uniqueness constraint, in the same database transaction as the reporting update. On replay, the application recognizes the already-applied event and avoids adding the order amount twice. The transaction matters: separately recording the event ID and changing the totals creates another failure window.

Committing before processing creates the opposite problem. If the consumer commits 45 and crashes before finishing offsets 42 through 44, a replacement normally starts at 45 and skips that unfinished work. The records may still exist, but ordinary recovery will not revisit them.

A commit also covers progress through a partition. If 43 fails while 44 succeeds, committing 45 skips the failure on restart. Applications that process records concurrently must track unfinished work before moving the saved position forward.

The client can recover connections and retry eligible network operations. It cannot decide whether repeating a database update is safe. Even explicit commits after processing leave a gap between two separate systems; they do not make the database write and Kafka commit one atomic operation.

Pull-based reading lets the application control how quickly it accepts work, but it still needs to remain an active member of its group.

The client sends heartbeats, periodic messages that let the coordinator detect a missing member. It also monitors the interval between calls to poll(). These checks address different failures: a disconnected process and a process that remains connected but stops returning to its consumption loop.

Suppose one poll returns 100 records and each database operation takes 50 milliseconds. Sequential processing takes roughly five seconds before allowing for other work. If database latency rises to two seconds per record, the same batch takes roughly 200 seconds. The broker can be healthy throughout that slowdown.

The max.poll.interval.ms setting limits the permitted gap between polls during group management. Exceeding it can lead to lost partition ownership and failed commits. The exact reassignment timing depends on the protocol and membership settings, so an application should not assume a timeout instantly stops all old processing.

A rebalance changes partition assignments among group members. When work moves elsewhere, the old consumer’s application must stop or reconcile work for the affected partitions. Otherwise, database actions from the old process can overlap with actions from the new owner.

Reducing records per poll can help keep sequential processing within a predictable time budget. For workloads with slow or variable operations, worker threads may be useful, but they add responsibility: bound the work queue, preserve any required partition order, and commit only completed progress.

The consumer’s pause() and resume() methods let the application temporarily withhold delivery from selected assigned partitions. It can keep polling for group activity while workers catch up. Pausing does not undo records the application has already handed to workers or release the partition assignment.

If producers continuously outpace processing, the backlog grows in Kafka or in application memory. More memory only delays the problem. The service needs enough processing capacity, a way to reduce incoming work, or an explicit policy for handling the backlog before retention removes needed records.

The Java KafkaConsumer is not thread-safe. A straightforward service gives one thread ownership of the consumer and its poll loop. That thread can process records itself or coordinate a limited amount of work on other threads.

Sharing one consumer freely across worker threads makes it difficult to reason about assignments, positions, and commits. Even when workers handle database operations, keep consumer API calls and progress decisions under a clear owner. wakeup() is a supported exception that another thread can use to interrupt a blocked consumer operation during shutdown.

For the reporting service, an orderly shutdown stops accepting more records, accounts for in-progress database work, commits offsets only for completed work and only for partitions it still owns, and closes the consumer. Closing releases client resources and allows group departure; it does not finish application work automatically.

A shutdown deadline may leave some records unfinished or some commits unconfirmed. Recovery must still tolerate replay. The same distinction used during normal operation applies at shutdown: records returned, work completed, and progress saved can describe different points in the stream.

The consumer discovers brokers, obtains partition assignments, fetches batches, and deserializes records for the application through poll(). It communicates with partition-serving brokers for data and with a group coordinator for membership and saved progress.

Fetched data, returned records, completed application work, and committed offsets advance separately. A reliable application keeps those boundaries clear, controls outstanding work, respects partition ownership, and makes recovery safe when processing repeats.