Definition
- Separation of concerns – the scheduler owns timing; workers own execution; a queue decouples them.
- Atomic transition – the scheduler moves a job from SCHEDULED to QUEUED in one indivisible step.
- Visibility timeout – a claimed job becomes invisible for a lease duration; if the worker crashes before ack, it reappears.
- Retries – backoff + jitter, max attempts, then dead-letter queue.
- Cancellation – worker re-reads the job's authoritative state before executing; if CANCELLED, skip.
- Hot-bucket sharding – hash(jobId) % N partitions a busy bucket so N workers drain in parallel.
The scheduling clerk watches the clock and drops due tasks into the out-tray. Processing clerks pick from the tray. A task on loan to a clerk comes back if the clerk faints (visibility timeout). Repeated failures go to a problem tray (DLQ). Before starting, the clerk double-checks the task was not cancelled. A flood of tasks opens several numbered trays to spread the load.
Understanding the Concepts
Why separate scheduler from workers?
The scheduler is lightweight and timing-focused (usually one active + standbys). Workers are heavy, variable, and need many parallel instances. A queue decouples their pace and isolates failure.
package com.bank.scheduler;
public class JobScheduler {
private final SortedSetScheduler sorted;
private final JobQueue queue;
public void tick() {
// Atomic claim: fetch due jobs and remove from sorted set
List<String> dueJobs = sorted.claimDueJobs();
for (String jobId : dueJobs) {
queue.enqueue(jobId); // workers consume at their own pace
}
}
}
How does the atomic SCHEDULED→QUEUED transition work?
-- Lua: atomic fetch + remove + enqueue
local due = redis.call('ZRANGEBYSCORE', 'due_jobs', '-inf', ARGV[1], 'LIMIT', 0, 100)
for _, job in ipairs(due) do
redis.call('ZREM', 'due_jobs', job)
redis.call('LPUSH', 'job_queue', job)
end
return #due
-- Even if two scheduler instances run, each job is enqueued exactly once
How does a visibility timeout handle worker crashes?
public class WorkerLoop {
public void run() {
while (true) {
String jobId = queue.claim(Duration.ofMinutes(5)); // invisible for 5 min
try {
process(jobId);
queue.ack(jobId); // done → remove permanently
} catch (Exception e) {
// Crash or failure → lease expires → job reappears for re-delivery
log.error("Job {} failed, will be retried", jobId, e);
}
}
}
}
// At-least-once: job may run more than once → workers must be idempotent
How do retries work with backoff and DLQ?
public void retryOrDlq(String jobId, int attempt) {
if (attempt >= MAX_ATTEMPTS) {
dlqQueue.enqueue(jobId); // permanently broken → investigate
return;
}
long delay = (long)(Math.pow(2, attempt) * 1000 + rng.nextInt(500)); // backoff + jitter
sorted.schedule(jobId, System.currentTimeMillis() + delay); // re-schedule with delay
}
How does cancellation work?
public void process(String jobId) {
Job job = jobStore.findById(jobId);
if (job.getStatus() == Status.CANCELLED) {
queue.ack(jobId); // remove from queue, skip execution
return;
}
// Do the actual work
}
// The queue cannot un-enqueue a message, so the cancellation signal lives in the job store
How do you shard hot time-buckets?
int N = 8;
int shard = Math.abs(jobId.hashCode()) % N;
String key = "jobs:2026-10-24T00:00:shard-" + shard;
// N scheduler workers each drain one shard in parallel
// A given job always lands in the same shard (deterministic)
What are common mistakes?
// WRONG: workers are NOT idempotent
// Crash → re-delivery → double-charge customer!
// CORRECT: dedup by job ID
if (processedJobs.existsById(jobId)) return; // already done
processedJobs.save(jobId);
What should a senior engineer know?
- The whole design assumes idempotent workers because of at-least-once delivery.
- Monitor queue depth, job age, retry counts, and DLQ volume.
- For ordered jobs within an entity, route them to the same shard by entity ID.
- Long-running jobs must extend the lease (heartbeat) to avoid re-delivery while still running.
Visual Overview
Scheduler-Worker Architecture
┌─────────────┐ atomic claim ┌──────────┐ enqueue ┌──────────┐
│ Sorted Set │ ──────────────►│Scheduler │ ────────► │ Queue │
│ (due jobs) │ Lua: ZRANGE │ (timing) │ │ │
└─────────────┘ + ZREM └──────────┘ └────┬─────┘
│ dequeue
┌────────▼────────┐
│ Worker 1 │
│ Worker 2 │
│ Worker N │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ ack (done) │ crash/fail │
│ → remove │ → lease expires
│ │ → re-delivered
└──────────────┴──────────────┘
Retry Path
fail → backoff + jitter → re-schedule
max attempts → Dead Letter Queue (investigate)
Key Takeaway
Separating the scheduler from the workers means the scheduler owns timing and atomically moves due jobs from SCHEDULED to QUEUED, while workers own execution and consume from a queue at their own pace. The queue decouples their rates and isolates failure: a worker crash does not stop the scheduler, and a scheduler restart does not lose in-flight work.
Worker crashes are handled by a visibility timeout: a claimed job is invisible for a bounded lease, and if the worker dies before acknowledging, the lease expires and the job is re-delivered. This gives at-least-once delivery, so workers must be idempotent. Retries use backoff with jitter, and after a max attempt count the job routes to a dead-letter queue. Cancellation is honored by checking the job's authoritative state just before executing.
At National Bank of India, Pankaj built this for reminders, settlement retries, and hold expirations. Sharding the midnight bucket by hash(jobId) % 8 spread the flood across parallel workers, the visibility timeout recovered jobs from crashed workers, and backoff with a DLQ tamed a flaky downstream.
Remember one sentence: the scheduler decides when and claims atomically, workers decide how via a queue, a visibility timeout recovers crashes, backoff plus a DLQ bounds retries, a state check honors cancellation, and sharding spreads hot buckets.
---