Design a Distributed Job Scheduler (Cron at Scale)

Every company has “run this job every night at 2 AM” — until one box dies and nobody notices, or ten boxes all fire the same cron. A distributed job scheduler is the grown-up version of crontab: durable schedules, many workers, retries, and observability. Interviewers like it because it mixes message queues, locking, and clock skew without needing a flashy consumer product.

Use the framework: one-shot vs recurring, max job runtime, and whether “at-least-once” execution is acceptable (almost always yes if jobs are idempotent).

Functional requirements

• Register a job: schedule (cron or one-shot timestamp), payload/handler name, owner.
• Execute the job when due across a fleet of workers.
• Retry with backoff on failure; dead-letter after N attempts.
• Pause, resume, cancel, and list job history.
• Optional: dependencies (job B after A succeeds) as a stretch goal.

Non-functional requirements

• No single point of failure — scheduler and workers must survive node loss.
• At-least-once delivery of “run” signals; jobs must be idempotent.
• Bounded delay: fire within seconds of the scheduled time under normal load.
• Fairness: one noisy tenant cannot starve others.

Capacity

Assume 10M registered schedules, 100K firings per minute at peak (think fan-out digests). Workers are CPU/IO bound on the job itself; the scheduler’s job is to enqueue work, not do the work. Separate control plane (when to run) from data plane (running the handler).

Architecture

1. API — create/update/delete schedules; auth via API gateway.
2. Job store — PostgreSQL or Cassandra: schedule definition + next_run_at.
3. Dispatcher — polls or listens for due jobs, claims them, pushes to queue.
4. Queue — Kafka / SQS with job_run_id payloads.
5. Workers — pull jobs, execute, report success/fail.
6. History / metrics — run logs, latency, failure rates.

Why not every worker runs crontab?

Local cron duplicates and drifts. Central schedule + distributed workers gives one source of truth. Classic anti-pattern: “deploy the same cron on 20 pods.”

Claiming due jobs (avoid double fire)

Dispatcher selects rows where `next_run_at <= now() AND status = scheduled`. Claim with optimistic locking: `UPDATE … SET status = running, locked_by = dispatcher_id WHERE id = ? AND status = scheduled`. Only one updater wins. Alternatively Redis `SET job:{id}:lock NX EX 60` before enqueue.

• After claim, enqueue message with unique run_id.
• On success: write history, compute next_run_at from cron, set status = scheduled.
• On failure: increment attempts, set next_run_at = now + backoff, or move to DLQ.
• Lease timeout: if worker dies, lock expires and job becomes claimable again.

Cron and next_run_at

Store cron expression + timezone. After each successful run, compute next fire time with a cron library. Index `next_run_at` so dispatchers can `SELECT … ORDER BY next_run_at LIMIT N FOR UPDATE SKIP LOCKED` (Postgres) — excellent interview phrase. SKIP LOCKED lets multiple dispatchers poll without blocking each other.

Thundering herd

Midnight UTC is when everyone schedules “daily reports.” Spread with jitter: `next_run_at += random(0, 60s)`. Shard schedules by `hash(job_id) % num_dispatcher_partitions` so load spreads. Rate-limit enqueue per tenant.

One-shot vs recurring

One-shot: delete or mark completed after success. Recurring: always advance next_run_at. Missed windows (dispatcher down for an hour): either catch up (run once for missed) or skip to next — product decision; say both options aloud.

Storage choice

PostgreSQL is enough for millions of schedules with a good `next_run_at` index. At extreme scale, shard by tenant or use a calendar queue in Redis sorted sets (`ZADD due_jobs score=timestamp member=job_id`) similar to leaderboard ZSETs — score is fire time. Redis is fast but needs persistence story (AOF/RDB) or dual-write to SQL for durability.

Observability

• Metrics: runs started, succeeded, failed, lag (now − scheduled_time).
• Trace run_id across dispatcher → queue → worker.
• Alert when lag > threshold or DLQ depth grows.
• Audit who changed a cron — compliance loves this.

Worked example

1. Host registers `0 2 * * *` nightly billing job, timezone Asia/Kolkata.
2. At 02:00 IST, dispatcher claims job, enqueues run_id R1.
3. Worker executes; billing API called with idempotency key R1.
4. Success → next_run_at set to tomorrow 02:00 + 12s jitter.
5. If worker crashes mid-run, lease expires; another worker may retry — billing ignores duplicate R1.

Multi-tenant isolation

SaaS schedulers need per-tenant quotas: max jobs, max concurrent runs, enqueue rate. Separate Kafka partitions or SQS queues per tier (free vs paid) so noisy neighbors cannot block VIP digests. Store `tenant_id` on every schedule row and filter dispatcher polls with fair round-robin across tenants when backlog is deep.

Comparison with workflow engines

Temporal / Cadence / Step Functions add durable workflows with signals and timers. A plain job scheduler is cron + queue. If the interviewer asks for “saga across services,” escalate to a workflow engine; do not bolt a state machine into crontab. Knowing when to stop is part of the answer.

Backfill and manual runs

Operators need “run now” and “re-run failed run_id.” Treat manual triggers as one-shot enqueues with a new run_id so idempotency keys stay unique. Audit every manual fire. For data pipelines, expose a backfill API that enqueues a date-range of logical runs rather than changing the cron definition.

Interview summary

Draw API → job store → dispatcher → queue → workers. Emphasize claim/lock, at-least-once + idempotent handlers, SKIP LOCKED or Redis locks, and jitter for midnight herds. Tie retries to notification delivery patterns. That is a complete scheduler answer.