Dead Letter Queues
Practice this topic in a realistic system design interview
In this chapter, we'll look at dead letter queues, why message-based systems need them, and how to design, monitor, and replay them correctly.
Imagine an order processing service that reads messages from a queue. Each message represents an order, and the consumer validates it, reserves inventory, and updates the database. Most messages are processed without any issues.
But one day, a message arrives with a product ID that doesn't exist. Maybe a bug in another service generated it, or the product was deleted before the order could be processed.
The consumer tries to process the message and hits an error. Because the message was never acknowledged, the queue delivers it again. The consumer fails again, and the same cycle continues. With at-least-once delivery, the message can keep coming back unless something explicitly stops it.
A message that consistently fails processing is called a poison message. It creates several problems:
- It wastes worker time that could be spent processing valid messages.
- It fills your logs with repeated errors, making other issues harder to spot.
- It can block other messages. In systems where ordering matters, such as a Kafka partition or an SQS FIFO message group, it can stop later messages from being processed.
In other words, a single bad message can hold up an entire stream of healthy messages.
One option is to simply discard the message after a few failed attempts. But then you risk losing important data. That message might represent a real customer's order that still needs to be fulfilled.
So we need a way to get the problem message out of the way without losing it.
1. What Is a Dead Letter Queue?
A dead letter queue, or DLQ, is a separate queue where messages go after they fail processing too many times.
Instead of retrying the poison message forever, or deleting it, the system moves it into the DLQ. The main queue continues processing the rest of the messages normally.
The failed message sits in the DLQ, where engineers can inspect it, understand why it failed, fix the problem, and send it back for processing.
You can think of the DLQ as a quarantine area. It keeps one bad message from affecting everything else, while making sure nothing is lost.