An online store needs to generate one invoice for each accepted order. It also needs to share order events with analytics, fulfillment, and reporting. Both jobs involve sending messages, but they need different ways to track and process them.

A work queue tracks tasks workers still need to complete. Kafka keeps a stream of records that several applications can read independently, each at its own pace.

In this chapter, we’ll compare these models through the store’s invoice and order events. We’ll look at replay, sharing work, ordering, and recovering from failures.

Here, a traditional message queue delivers tasks to workers and removes them once workers acknowledge them. We’ll assume the queue stores messages durably, messages do not expire, and workers acknowledge them only after processing. RabbitMQ queues are one example, though the details vary by product and queue type.

A worker is a consumer that performs a task. In our examples, it sends an acknowledgment after finishing, telling the queue it can remove the message. The broker trusts that signal; it does not check whether the worker actually created the invoice.

For Kafka, we’ll use regular consumer groups and topics that delete old records based on time or size limits. Reading a record or saving a group’s progress does not remove it from the topic.

Products can offer more than one model. RabbitMQ also has Streams, which keep messages for replay. Kafka 4.3 supports share groups, where consumers can share a partition and acknowledge individual records. Here, we’ll focus on traditional queues and Kafka’s regular consumer groups.

Start with what the application needs: tracking unfinished tasks, keeping events for independent readers, or both.

Suppose the store sends a GenerateInvoice task for order ord-1042, with the task ID invoice-1042. One worker should create that invoice. More workers let the store handle other orders at the same time.

Here’s how the task moves through a queue with manual acknowledgments:

The task stays unacknowledged while the worker processes it. If the connection closes before the broker receives an acknowledgment, the queue can deliver it again. After acknowledgment, consumers can no longer read it from that queue, though disk cleanup may happen later.

Kafka stores the record separately from each group’s progress. Suppose an OrderPlaced event with key ord-1042 is in partition 0 of orders.placed at offset 42. After processing it and all earlier records, a consumer can commit 43 as its next restart position. The record at offset 42 remains until the topic’s cleanup policy removes it.

Once the worker acknowledges the invoice task, the queue considers it finished. Kafka can still serve the order event to another group or to a consumer replaying earlier records.

Neither model guarantees that unread data stays available forever. Queues can have expiration and capacity limits, and Kafka can delete records before a slow consumer reads them. Plan storage and retention around the outages and backlogs your application needs to handle.

Two workers on the same queue share its tasks. Normally, each task goes to one worker, though the queue may deliver it again after a failure. This lets several processes help with the same job.

Analytics and fulfillment have a different need: both must receive every order event. Putting them on the same work queue would split the events between them, so each would miss some orders.

A queue-based broker can handle this with fan-out, where one publication reaches several destinations. In RabbitMQ, an exchange routes messages to queues using rules called bindings. Analytics and fulfillment each get a queue and acknowledge their own messages.

Kafka gives each application its own consumer group. The diagram compares the two setups. The arrows show where events go, without showing how clients fetch or receive them.

Both setups let several applications receive the same events. Queues track each application’s outstanding messages separately. Kafka stores a shared topic and tracks each group’s position in it.

If reporting joins tomorrow, a new queue you bind to the exchange receives matching events that producers publish from then on. It does not automatically get yesterday’s acknowledged messages. A new Kafka group can start at an earlier offset and read records that are still available. You need to choose that starting position; creating a group does not automatically make it read from the beginning.

Suppose a bug left discounted orders out of yesterday’s totals. The team fixes the calculation and needs to rebuild the report.

With a traditional queue, the acknowledged messages are gone from the queue. Rebuilding the report needs another source, such as an event archive or the order database. The queue does not keep a readable history of completed tasks.

With Kafka, a separate consumer group can reread yesterday’s records if they are still available. It can write corrected totals to a new table while the live reporting group keeps processing new events.

Replay also depends on how the calculation works. If it looks up a product’s current price, processing yesterday’s order today may produce a different total. Keeping the event does not preserve every external value the calculation used. The data and code need to support the result you want to rebuild.

Cleanup rules matter too. Our examples delete records based on time or size. Kafka also supports log compaction, which can remove older records for a key while keeping newer state. A compacted topic may not contain every event producers published.

A work queue distributes tasks among workers. One worker can finish a simple invoice while another is still handling a slower one. The broker distributes tasks according to its delivery rules and the number of unacknowledged messages each worker can hold.

A regular Kafka consumer group shares work by assigning partitions. Once assignments settle, each partition has one assigned consumer in the group. Here’s a topic with two partitions and a group with three consumers:

The third consumer has no partition to read. It can take over work if assignments change, but adding consumers alone does not split a partition between group members. A consumer can process records in parallel internally, but the application then needs to manage ordering and track progress carefully.

Kafka preserves record order within each partition. There is no single order across the whole topic. Even within a partition, if a consumer sends offsets 42 and 43 to different threads, the work for 43 may finish first. Reading in order does not guarantee that database updates or other actions finish in order.

Queues have a similar concern. Workers can finish tasks in a different order from the order the queue delivers them. Redeliveries and message priorities can also change the order workers receive them.

For invoices from unrelated orders, completion order may not matter. For a series of updates to the same order, it may be essential. Decide which operations must stay in order, then preserve that order throughout processing.

Suppose a worker creates invoice invoice-1042, then crashes before telling the messaging system it has finished.

The queue may deliver the task again because it never received an acknowledgment. A Kafka consumer may also read the record again when it resumes from its last committed offset. In both cases, the application needs a way to recognize that the invoice already exists.

Both models can support at-least-once processing: recovery may attempt the same work more than once. Reporting completion before doing the work creates the opposite risk. A crash could leave the task unfinished even though the messaging system considers it complete.

Invoice creation should therefore be idempotent: repeating the request should have the same business result as doing it once. A database uniqueness constraint on the order ID can prevent a second invoice row for that order. If creation calls an external service, the service may also need to recognize a stable request ID and avoid repeating the action. Including an ID in the message is only useful if something checks it.

A Kafka offset commit saves a position in a partition. Suppose processing fails at offset 42 but succeeds at 43. Committing 44 tells a restarting consumer to skip both records, including the unfinished work at 42. A sequential consumer can pause and retry 42. A consumer processing in parallel needs to track unfinished records before advancing its committed offset.

Some queues let workers reject individual messages and put them back on the queue. Kafka consumers often handle retries in application code, sometimes with extra topics. Either way, decide what to do with messages that keep failing. Setting one aside lets later work continue, but may break an ordering requirement.

Publisher confirmations and producer acknowledgments tell the sender about the write to the messaging system. Consumer acknowledgments and offset commits report the reader’s progress. Neither proves that a payment, email, or database update happened exactly once. Kafka transactions can coordinate Kafka records and offsets in supported workflows, but external actions need their own coordination.

Here’s how the two models compare. Individual products may offer features beyond these basics.

For the store’s invoices, a work queue fits the need to distribute independent tasks and track completion. For order events, Kafka fits when several applications need their own progress and access to earlier records. A system can use both if the benefit is worth the work of running and connecting them.

Neither model wins on performance in every situation. Message size, batching, replication, consumer behavior, and deployment all affect throughput, latency, storage costs, and recovery time. Compare them using the workload and durability requirements your application actually has.

Push versus pull is also an incomplete comparison. Kafka consumers fetch records, while queue products offer different delivery options. Focus first on how long data stays available and whether consumers share work by message or by partition.

A traditional queue tracks unfinished messages and removes them after acknowledgment. Kafka keeps records according to its cleanup policy and tracks where each consumer group has reached. This lets applications read and replay available records independently.

Both models can share work and serve several applications. Choose based on how you need to distribute tasks, keep history, replay records, and preserve order. In either case, the application needs to handle retries and save progress in a way that reflects completed work.