Design a Distributed Job Scheduler
What is a Job Scheduler?
A Job Scheduler is a system that manages the execution of tasks at specified times or intervals. Unlike a task queue where jobs are executed immediately upon submission, a job scheduler allows you to define when a job should run, whether that's a one-time execution in the future or a recurring pattern like "every day at midnight."
Loading simulation...
The core idea is to decouple the job definition from its execution. Users define what should run and when, and the scheduler ensures it happens reliably, even across distributed systems where multiple servers need to coordinate to prevent duplicate executions.
Popular Examples: Linux Cron, Kubernetes CronJobs, Apache Airflow
This problem tests your understanding of distributed coordination, time-based systems, fault tolerance, and trade-offs between consistency and availability.
In this article, we will explore the high-level design of a distributed job scheduler.
Let's start by clarifying the requirements.