In this article, we will explore the high-level design of a distributed task queue.

Let's start by clarifying the requirements.

Before starting the design, it's important to ask thoughtful questions to uncover hidden assumptions, clarify ambiguities, and define the system's scope more precisely.

Here is an example of how a discussion between the candidate and the interviewer might unfold:

After gathering the details, we can summarize the key system requirements.

To understand the scale of our system, let's make some reasonable assumptions.

Task Submissions (Writes)

Task Status Queries (Reads)

Worker Throughput

Storage (30 days retention)

Each task stores:

Total ≈ 1.2 KB per task

Monthly storage: 100M tasks/day * 30 days * 1.2 KB ≈ 3.6 TB for task metadata.

The distributed task queue needs APIs for task submission, status tracking, and management. Below are the core APIs required for basic functionality.

1. Submit Task

Endpoint: POST /tasks

Submits a new task to the queue for execution.

Request Parameters

Sample Response

Error Cases

2. Get Task Status

Endpoint: GET /tasks/{task_id}

Retrieves the current status and details of a task.

Sample Response

Status Values

Error Cases

3. Cancel Task

Endpoint: DELETE /tasks/{task_id}

Cancels a pending or scheduled task. Running tasks cannot be cancelled.

Sample Response

Error Cases

4. Internal: Fetch Tasks (Worker API)

Endpoint: POST /internal/tasks/fetch

Workers call this endpoint to fetch tasks for execution. This is an internal API not exposed to clients.

Request Parameters

Sample Response

At a high level, our system must satisfy three core requirements:

The system has an interesting asymmetry: task submissions are relatively lightweight writes, but task distribution requires careful coordination to ensure exactly one worker picks up each task.

Note: Instead of presenting the full architecture at once, we'll build it incrementally by addressing one requirement at a time. This approach is easier to follow and mirrors how you would explain the design in an interview.

Clients need to submit tasks reliably. Even if internal components are temporarily unavailable, we should accept and durably store the task.

Components Needed

API Gateway

The entry point for all client requests. Handles authentication, rate limiting, and request routing.

It authenticates and authorizes clients, applies rate limiting per client or tenant, and routes requests to the appropriate service.

Task Service

Handles all task-related operations: submission, status queries, and cancellation.

It validates task payloads against registered schemas, generates unique task IDs, persists task metadata to the database, and enqueues tasks for processing.

Task Database

Stores task metadata, status, and results. This is the source of truth for task state.

It durably stores all task information, supports efficient queries by task ID and status, and handles high write throughput for task submissions.

Flow: Submitting a Task

The three components form a simple chain: the client sends the request, the Gateway validates it, and the Task Service writes the task to the database before returning the task ID.

At this point, the task is durably stored but not yet available for workers. We need a mechanism to make it available for execution.

Workers need to efficiently discover and claim tasks. This is the most challenging part of the design because we must ensure each task is processed by exactly one worker with no duplicates, tasks are distributed fairly across workers, and high-priority tasks are processed before low-priority ones.

Additional Components Needed

Message Queue

A distributed queue that holds tasks ready for execution. Workers pull tasks from this queue.

It stores tasks until workers are ready to process them, supports multiple priority levels through separate queues per priority, and provides a visibility timeout to handle worker failures.

Technology choices: Redis, Amazon SQS, RabbitMQ, or Apache Kafka.

Scheduler Service

Handles delayed tasks and periodic scheduling. Moves tasks from "scheduled" state to the message queue when their time arrives.

It tracks tasks scheduled for future execution, moves delayed tasks to the queue at the scheduled time, and handles recurring or cron-style tasks where supported.

Flow: Making Tasks Available

Two paths lead to the Message Queue: the Task Service enqueues immediate tasks directly, while the Scheduler polls the database for delayed tasks whose scheduled time has arrived and feeds them into the same queue.

For immediate tasks

For delayed tasks

Workers must reliably execute tasks and handle failures gracefully.

Additional Components Needed

Workers

Stateless processes that fetch tasks from the queue, execute them, and report results.

Each worker fetches tasks from the message queue, executes task logic based on task type, reports success or failure back to the Task Service, and handles graceful shutdown without losing tasks.

Dead Letter Queue (DLQ)

A separate queue for tasks that have failed permanently (exceeded retry limits).

It stores failed tasks for manual inspection, preserves full task context for debugging, and allows manual retry or deletion.

Flow: Executing a Task

The solid arrows trace the success path from queue to worker to status update, while the dashed arrow marks the failure path that routes a task to the Dead Letter Queue once retries are exhausted.

Here is the complete architecture combining all requirements:

Core Components Summary

5.1 SQL vs NoSQL

To choose the right database for our needs, let's consider some factors:

Given these points, a SQL database like PostgreSQL is a solid choice. It offers strong consistency guarantees for status updates, efficient indexing for status-based queries, and mature tooling for operations and monitoring.

For very high scale (billions of tasks), a NoSQL database like Cassandra or DynamoDB may be more appropriate, with the trade-off of eventual consistency.

Hybrid approach: Use PostgreSQL for task metadata and Redis for the actual queuing mechanism. This combines the durability of SQL with the speed of in-memory queues.

5.2 Database Schema

1. Tasks Table

Stores all task metadata and state.

Indexes:

2. Task History Table

Stores execution attempts for debugging and auditing.

3. Dead Letter Queue Table

Stores tasks that have permanently failed.

Now that we have the high-level architecture and database schema in place, let's dive deeper into some critical design choices.

How workers receive tasks is fundamental to the system's performance and reliability. The right strategy depends on your scale, latency requirements, and infrastructure.

Approach 1: Pull-Based (Polling)

Workers actively poll the queue for new tasks at regular intervals.

How It Works

Optimization: Long Polling

Instead of short polls with sleep intervals, use long polling: the worker's fetch request blocks until either a task is available or a timeout expires.

This reduces empty polls while keeping latency low.

Approach 2: Push-Based (Event-Driven)

The queue actively pushes tasks to workers as they become available.

How It Works

Approach 3: Hybrid (Recommended)

Combine the best of both approaches: use push notifications to wake workers, but let workers pull the actual tasks.

How It Works

This provides the low latency of push with the backpressure control of pull.

Summary and Recommendation

Recommendation: Start with long polling for simplicity. Move to a hybrid approach if you need sub-second latency at scale.

Message delivery semantics determine what happens when failures occur. This is one of the most important design decisions for a task queue.

At-Most-Once Delivery

Each task is delivered to a worker at most one time. If the worker crashes, the task is lost.

How It Works

When to Use

At-Least-Once Delivery

Each task is guaranteed to be delivered at least once, but may be delivered multiple times.

How It Works

The Visibility Timeout

The visibility timeout is critical for at-least-once delivery:

Best practice: Set the visibility timeout to 2-3x the expected task duration. For variable-duration tasks, allow workers to extend their lease.

Handling Duplicates

Since tasks may be delivered multiple times, your workers must be idempotent. Common strategies:

Exactly-Once Delivery

Each task is processed exactly once, regardless of failures. This is the holy grail but is extremely difficult to achieve in distributed systems.

Why It's Hard

True exactly-once delivery requires coordination between the queue and the processing logic, which typically means:

In practice, "exactly-once" is usually implemented as "at-least-once delivery with idempotent processing."

Recommendation

Recommendation: Use at-least-once delivery with idempotent task handlers. This provides strong guarantees without the complexity of true exactly-once semantics.

Failures are inevitable in distributed systems. A well-designed retry strategy ensures tasks eventually succeed while avoiding infinite loops and system overload.

Types of Failures

Retry Strategy

1. Exponential Backoff

Increase the delay between retries exponentially to avoid overwhelming failing dependencies.

Formula: delay = base_delay * (multiplier ^ attempt)

2. Jitter

Add randomness to prevent synchronized retries from many failed tasks hitting the system simultaneously.

3. Maximum Retries

Set a limit to prevent infinite retry loops. Common values are 3-5 retries.

4. Retry Classification

Not all errors should be retried. Classify errors and handle them appropriately:

Dead Letter Queue (DLQ)

Tasks that exhaust all retries are moved to a Dead Letter Queue for manual inspection.

DLQ Best Practices

Poison Pill Detection

Some tasks can crash workers (out-of-memory, infinite loops). Detect and isolate these:

When your queue handles different types of tasks, you need mechanisms to ensure important tasks are processed first while still making progress on lower-priority work.

Separate Queues per Priority

The simplest approach: maintain separate queues for each priority level.

Worker Assignment Strategies

Dedicated workers: Assign specific workers to each priority level.

Weighted polling: Workers poll high-priority queues more frequently.

Priority draining: Workers always drain higher-priority queues first.

Preventing Starvation

If high-priority tasks constantly arrive, low-priority tasks may never execute. Strategies to prevent this:

Task Affinity

Some tasks should be processed by specific workers (e.g., tasks for the same user should go to the same worker for caching benefits).

Consistent Hashing

Route tasks to workers based on a hash of the task's affinity key:

This ensures tasks for the same user usually go to the same worker, while still distributing load evenly.

As task volume grows, different components require different scaling approaches.

Scaling the Task Service

The Task Service is stateless and can be horizontally scaled behind a load balancer.

Auto-scaling triggers

Scaling the Message Queue

Different queue technologies scale differently:

Scaling Workers

Workers are the most common scaling target. Considerations:

Scaling the Database

The Task Database can become a bottleneck at high scale.

Read scaling: Add read replicas for status queries. Write scaling: Shard by task ID or tenant ID. Archive old data: Move completed tasks to cold storage after a retention period.

Multi-Region Deployment

For global availability, deploy the task queue in multiple regions:

While true exactly-once delivery is difficult, we can achieve effective exactly-once processing through idempotency.

Idempotency Keys

Clients provide a unique key with each task submission. The system uses this key to detect and reject duplicates.

Implementation

Worker-Side Idempotency

For tasks that modify external systems, the worker must ensure idempotency:

Deduplication Windows

For some use cases, you may want to deduplicate tasks within a time window (e.g., don't send more than one notification per user per hour).

Implementation