Design a Distributed Task Scheduler (like cron at scale)

• Distributed task scheduler : reliably runs jobs at or after a specified time across a cluster of workers.
• Lease-based claiming : a worker atomically claims a job with a time-limited lease; expired leases trigger re-dispatch.
• Deduplication : a stable execution id lets the system recognize a duplicate run and skip it.
• Dead-letter state : jobs that fail after max retries are moved aside for investigation.
• Multi-tenant isolation : per-tenant concurrency caps prevent one tenant from starving others.

How are jobs stored?

Each job record holds its id, due time, payload, tenant, state, and retry bookkeeping. A time-based index efficiently finds due jobs by querying for records where due_at is less than or equal to the current time. At National Bank of India, the job store holds thousands of scheduled tasks: nightly interest calculations, monthly statement generation, and settlement retries.

package com.bank.scheduler;

import java.util.Comparator;
import java.util.List;

public class JobStore {
    public List<Job> dueJobs(long now, int limit) {
        return store.values().stream()
            .filter(j -> j.state() == State.PENDING && j.dueAt() <= now)
            .sorted(Comparator.comparingLong(Job::dueAt))
            .limit(limit).toList();
    }
}

What does a job record look like?

A job record captures everything the system needs to execute, retry, and audit the task. The tenant field enables multi-tenant isolation. The attempt count and max retries drive the retry policy.

package com.bank.scheduler;

public record Job(String id, String tenantId, long dueAt, String payload,
                  State state, String leaseOwner, long leaseExpiry,
                  int attemptCount, int maxRetries) {}

public enum State { PENDING, RUNNING, COMPLETED, FAILED, DEAD_LETTER }

How does lease-based claiming work?

A worker claims a job atomically by updating its state from PENDING to RUNNING with a lease expiry timestamp. If the worker crashes, the lease expires and a reaper thread re-dispatches the job. The atomic claim prevents two workers from executing the same job simultaneously.

package com.bank.scheduler;

public class LeaseManager {
    public boolean claim(String jobId, String workerId, long leaseMs) {
        int rows = jdbc.update(
            "UPDATE jobs SET state = 'RUNNING', lease_owner = ?, " +
            "lease_expiry = ? WHERE id = ? AND state = 'PENDING'",
            workerId, System.currentTimeMillis() + leaseMs, jobId);
        return rows == 1;  // atomic claim
    }
    public void releaseExpiredLeases(long now) {
        jdbc.update(
            "UPDATE jobs SET state = 'PENDING', lease_owner = NULL " +
            "WHERE state = 'RUNNING' AND lease_expiry < ?", now);
    }
}

How does deduplication prevent double execution?

A stable execution id lets the system recognize that the job's effect was already applied. Even if a job runs twice (worker A crashes after completing but before releasing the lease, worker B picks it up), the idempotent execution checks the id and skips the duplicate. This is critical for interest calculation: running it twice would double the credited interest.

package com.bank.scheduler;

public class IdempotentExecutor {
    public void execute(Job job) {
        String execId = job.id() + ":" + job.attemptCount();
        if (executionLog.exists(execId)) return;  // already applied
        runPayload(job.payload());
        executionLog.record(execId);
    }
}

How does retry with backoff work?

Failed jobs are re-scheduled after a backoff delay that doubles with each attempt. After max attempts, the job moves to dead-letter state for manual investigation. National Bank of India's operations team monitors the dead-letter queue to catch persistent failures like unreachable partner bank APIs.

package com.bank.scheduler;

public class RetryPolicy {
    public long nextDueAt(Job job) {
        long backoffMs = 1000L * (long) Math.pow(2, job.attemptCount());
        return System.currentTimeMillis() + backoffMs;
    }
    public boolean shouldRetry(Job job) {
        return job.attemptCount() < job.maxRetries();
    }
}

How does multi-tenant isolation work?

Per-tenant concurrency caps ensure one tenant's flood of jobs does not starve others. Before dispatching a job, the scheduler checks how many jobs that tenant currently has running. If the cap is reached, the job stays in the queue until a slot opens.

How do you handle recurring jobs?

After completion, compute the next run time from the cron expression and insert a new job record. The completed job stays in the database for audit purposes.

How do you tune the lease?

Long enough for legitimate execution (e.g., interest calculation takes 30 seconds, so the lease is 2 minutes). Short enough for quick recovery from dead workers. Completion must be recorded atomically with releasing the lease to prevent a race where the lease expires and a second worker picks up a job that already succeeded.

What should a senior engineer know?

A senior engineer monitors lease expiry rates (an indicator of worker health), job queue depth per tenant, and designs completion recording atomically with lease release. They also consider priority queues so critical jobs (like settlement) run before lower-priority ones (like report generation).