Design Distributed Job Scheduler
What is a Job Scheduler?
A job scheduler is a system that manages the execution of tasks at specified times or intervals. These jobs may include batch processing, data pipelines, report generation, or recurring background tasks such as sending reminders.
Examples include cron, Apache Airflow, and Kubernetes CronJobs.
In this article, we will walk through the process of designing a scalable distributed job scheduling service that can handle millions of tasks, and ensure high availability.
Let’s begin by clarifying the requirements.
1. Requirement Gathering
Before diving into the design, let’s outline the functional and non-functional requirements.
Functional Requirements:
- Users can submit one-time or periodic jobs for execution.
- Users can cancel the submitted jobs.
- The system should distribute jobs across multiple worker nodes for execution.
- The system should provide monitoring of job status (queued, running, completed, failed).
- The system should prevent the same job from being executed multiple times concurrently.
Non-Functional Requirements:
- Scalability: The system should be able to schedule and execute millions of jobs.
- High Availability: The system should be fault-tolerant with no single point of failure. If a worker node fails, the system should reschedule the job to other available nodes.
- Latency: Jobs should be scheduled and executed with minimal delay.
- Consistency: Job results should be consistent, ensuring that jobs are executed once (or with minimal duplication).
Additional Requirements (Out of Scope):
- Job prioritization: The system should support scheduling based on job priority.
- Job dependencies: The system should handle jobs with dependencies.