Definition
- @Scheduled is per-JVM -- each replica fires the job independently, so N instances means N duplicate executions.
- ShedLock -- a lightweight library that wraps
@SchedulerLockaround a scheduled method; each instance tries to acquire a named lock in a shared store (database table, Redis), and only the winner runs. - lockAtMostFor -- safety timeout that auto-releases the lock if the holder crashes, so future triggers are not blocked forever.
- lockAtLeastFor -- minimum hold time that guards against clock skew, preventing a fast job from being picked up again within the same window.
- Leader election -- one instance is elected leader (ZooKeeper, Kubernetes lease, Spring Cloud); only the leader runs scheduled jobs.
- Quartz clustering -- the Quartz scheduler in clustered mode with a
JDBCJobStoreon a shared database; row locking ensures each trigger fires on exactly one node.
National Bank of India runs four copies of its batch service for availability. At midnight all four try to generate statements, so every customer gets four copies. ShedLock is like a single shared sign-in sheet at the vault: whoever grabs the pen first does the job, the others see it is taken and walk away. Leader election is like appointing one branch as head office for all nightly tasks. Quartz clustering is like a central scheduling office with a shared diary that assigns each task to exactly one branch.
Understanding the Concepts
Why does @Scheduled fire on every instance?
@Scheduled is local to each JVM. Spring's scheduler lives inside each application instance and has no awareness of other instances. With four replicas, the midnight cron fires four times. For a banking job that writes data, like generating statements or accruing interest, this duplicates financial effects.
package com.bank.batch;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
public class StatementJob {
@Scheduled(cron = "0 0 0 * * *")
public void generateStatements() {
// Runs on EVERY instance -- 4 replicas = 4 duplicate statement runs
statementService.generateAll();
}
}
With 4 replicas at midnight:
Instance A: generateStatements() -- runs
Instance B: generateStatements() -- runs (duplicate!)
Instance C: generateStatements() -- runs (duplicate!)
Instance D: generateStatements() -- runs (duplicate!)
Result: every customer gets 4 identical statements
How does ShedLock solve the duplicate execution problem?
ShedLock wraps the scheduled method with a distributed lock. Before running, each instance tries to acquire a named lock in a shared store. Only the one that acquires it runs; the rest skip.
package com.bank.batch;
import net.javacrumbs.shedlock.spring.annotation.EnableSchedulerLock;
import net.javacrumbs.shedlock.spring.annotation.SchedulerLock;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
public class GuardedStatementJob {
@Scheduled(cron = "0 0 0 * * *")
@SchedulerLock(name = "nightlyStatements",
lockAtMostFor = "PT30M",
lockAtLeastFor = "PT5M")
public void generateStatements() {
// Runs on exactly ONE instance
statementService.generateAll();
}
}
With ShedLock at midnight:
Instance A: acquires lock 'nightlyStatements' -- runs
Instance B: lock already held -- skips
Instance C: lock already held -- skips
Instance D: lock already held -- skips
Result: exactly one statement run
How do you configure ShedLock with a database lock provider?
ShedLock needs a shared store. The simplest is a database table. Create the table, configure a LockProvider bean, and enable scheduler locking.
-- ShedLock table (PostgreSQL)
CREATE TABLE shedlock (
name VARCHAR(64) NOT NULL PRIMARY KEY,
lock_until TIMESTAMP NOT NULL,
locked_at TIMESTAMP NOT NULL,
locked_by VARCHAR(255) NOT NULL
);
package com.bank.config;
import net.javacrumbs.shedlock.core.LockProvider;
import net.javacrumbs.shedlock.provider.jdbctemplate.JdbcTemplateLockProvider;
import net.javacrumbs.shedlock.spring.annotation.EnableSchedulerLock;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import javax.sql.DataSource;
@Configuration
@EnableScheduling
@EnableSchedulerLock(defaultLockAtMostFor = "PT10M")
public class ShedLockConfig {
@Bean
public LockProvider lockProvider(DataSource dataSource) {
return new JdbcTemplateLockProvider(dataSource);
}
}
Expected behavior:
- shedlock table stores one row per job name
- Atomic INSERT/UPDATE ensures only one instance wins
- lock_until column auto-releases a crashed holder's lock
What do lockAtMostFor and lockAtLeastFor mean?
lockAtMostFor is the safety timeout. If the winning instance crashes mid-job, the lock would otherwise be held forever. After lockAtMostFor the lock auto-expires so a future trigger is not blocked.
lockAtLeastFor is the minimum hold. It prevents a job that finishes in milliseconds from being re-run by a second instance within the same trigger window due to clock skew between servers.
| Attribute | Purpose | Typical Value |
|-----------|---------|---------------|
| lockAtMostFor | Safety: auto-release after crash | Longer than the expected job duration |
| lockAtLeastFor | Guard: prevent re-run within same window | A few minutes |
@SchedulerLock(name = "interestAccrual",
lockAtMostFor = "PT2H", // job may run up to 2 hours
lockAtLeastFor = "PT10M") // hold at least 10 min
public void accrueInterest() {
// If the job finishes in 30 seconds, lock is still held for 10 minutes
// If the job crashes after 1 hour, lock auto-releases after 2 hours
}
Scenario 1 (fast completion):
- Job finishes in 30 seconds
- Lock held for 10 minutes (lockAtLeastFor)
- No second instance can re-run within this window
Scenario 2 (crash):
- Job crashes after 45 minutes
- Lock auto-releases after 2 hours (lockAtMostFor)
- Next trigger at 2:00 AM finds the lock expired and runs
How does leader election work for scheduled jobs?
Leader election designates a single instance as the leader. Only the leader executes scheduled jobs. If it dies, another instance takes over. The election backend can be ZooKeeper, Consul, or a Kubernetes lease.
package com.bank.batch;
import org.springframework.integration.leader.Context;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
public class LeaderOnlyJob {
private final Context leaderContext;
public LeaderOnlyJob(Context leaderContext) {
this.leaderContext = leaderContext;
}
@Scheduled(cron = "0 0 0 * * *")
public void generateStatements() {
if (!leaderContext.isLeader()) {
return; // only the elected leader runs
}
statementService.generateAll();
}
}
Expected behavior:
Instance A (leader): runs generateStatements()
Instance B (follower): isLeader() returns false -- skips
Instance C (follower): isLeader() returns false -- skips
-- If Instance A dies, Instance B elected leader, next trigger runs on B --
How does Quartz clustering ensure single execution?
Quartz in clustered mode uses a shared JDBCJobStore. All nodes share the Quartz tables, and row locking ensures each trigger fires on exactly one node. Beyond run-once, Quartz provides persistent jobs, misfire handling, and rich scheduling.
# application.properties
spring.quartz.job-store-type=jdbc
spring.quartz.properties.org.quartz.jobStore.isClustered=true
spring.quartz.properties.org.quartz.scheduler.instanceId=AUTO
spring.quartz.properties.org.quartz.jobStore.clusterCheckinInterval=20000
package com.bank.batch;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.springframework.stereotype.Component;
@Component
public class QuartzStatementJob implements Job {
@Override
public void execute(JobExecutionContext context) {
// Quartz guarantees this fires on exactly one node
statementService.generateAll();
}
}
Expected behavior:
- Quartz tables store triggers and job state
- Row-level locking assigns each trigger to one node
- If the assigned node is down, misfire handling runs it on another
- Job state persists across restarts
How do you choose between ShedLock, leader election, and Quartz?
Each approach suits a different complexity level. ShedLock is minimal for simple run-once needs. Leader election fits when one node should own all singleton work. Quartz is the richest option for persistence, misfire handling, and dynamic scheduling.
| Feature | ShedLock | Leader Election | Quartz Clustering |
|---------|----------|-----------------|-------------------|
| Setup complexity | Low (one table) | Medium (ZK/Consul) | High (11+ tables) |
| Run-once guarantee | Yes | Yes (leader only) | Yes (row locking) |
| Persistence | Lock only | No (leadership state) | Full (jobs, triggers) |
| Misfire handling | No | No | Yes (fire now, skip, etc.) |
| Dynamic rescheduling | No | No | Yes (rescheduleJob) |
| Best for | Simple cron jobs | Singleton responsibilities | Complex scheduling |
Decision flow:
- Simple run-once cron? --> ShedLock
- One node owns all periodic tasks? --> Leader election
- Need persistence, misfire, dynamic triggers? --> Quartz clustering
What are common mistakes with distributed scheduling?
The most common mistake is not guarding a scheduled job at all when running multiple instances, leading to duplicate executions. The second is setting lockAtMostFor too short, so the lock expires while the job is still running.
// WRONG: no guard -- runs on every instance
@Scheduled(cron = "0 0 0 * * *")
public void generateStatements() {
statementService.generateAll();
}
// WRONG: lockAtMostFor too short -- expires mid-job
@Scheduled(cron = "0 0 0 * * *")
@SchedulerLock(name = "statements", lockAtMostFor = "PT1M")
public void generateStatements() {
// Job takes 30 minutes, lock expires after 1 minute
// Another instance picks it up and runs in parallel!
}
// CORRECT: proper guard with safe timings
@Scheduled(cron = "0 0 0 * * *")
@SchedulerLock(name = "statements",
lockAtMostFor = "PT2H",
lockAtLeastFor = "PT5M")
public void generateStatements() {
statementService.generateAll();
}
Common mistakes:
1. No guard at all -- N instances = N duplicate runs
2. lockAtMostFor too short -- lock expires mid-job, parallel execution
3. lockAtMostFor too long -- crashed holder blocks the next trigger for hours
4. Not making the job idempotent -- edge cases cause duplicates
What should a senior engineer know about run-once scheduling?
A senior engineer knows that even run-once mechanisms have edge cases. A ShedLock lock can expire mid-job if the job runs longer than lockAtMostFor. A leadership flip during execution can briefly have two leaders. So the job must still be idempotent as the ultimate safety net. They also monitor and alert on both duplicate runs and missed runs, because a dead lock store prevents the job from running at all.
package com.bank.batch;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
import java.time.LocalDate;
@Service
public class IdempotentStatementService {
@Transactional
public void generateForAccount(String accountId) {
// IDEMPOTENT: upsert by account + period, so re-running never duplicates
if (statementRepo.existsByAccountAndPeriod(accountId, LocalDate.now().getMonth())) {
return; // already generated
}
Statement stmt = new Statement(accountId, LocalDate.now().getMonth());
stmt.setBalance(accountRepo.findBalance(accountId));
statementRepo.save(stmt);
}
}
Senior checklist:
1. Guard every scheduled job in a multi-instance deployment
2. Set lockAtMostFor longer than the expected job duration
3. Make the job idempotent -- the ultimate safety net
4. Monitor for both missed runs and duplicate runs
5. Keep the lock/leadership store highly available
Visual Overview
Midnight trigger fires on all 4 replicas
│
┌────┼────┬────────┬────────┐
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
A B C D
│ │ │ │
▼ ▼ ▼ ▼
┌────────────────────────────────┐
│ Shared Lock Store (DB/Redis) │
│ │
│ A: INSERT lock ── SUCCESS │
│ B: INSERT lock ── CONFLICT │
│ C: INSERT lock ── CONFLICT │
│ D: INSERT lock ── CONFLICT │
└────────────────────────────────┘
│
A runs the job B, C, D skip
ShedLock timeline:
──┬───────────────────────────────────────┬──► Time
│ │
Lock acquired lockAtMostFor
by Instance A (auto-release)
│◄── lockAtLeastFor ──►│
│ (minimum hold) │
│ │
│ Job completes │
│ here (fast) │
│ │
│ Lock still held │
│ until lockAtLeast │
Key Takeaway
A @Scheduled task is local to each JVM, so when you run multiple replicas for availability, the trigger fires inside every replica and the job runs once per instance. For a banking job that writes data, like generating statements or accruing interest, that means duplicated financial effects. ShedLock is the lightest fix: a shared lock store (usually a small database table) ensures only one instance acquires the named lock and runs the job, while the rest skip.
The two ShedLock timers matter and interviewers probe them. lockAtMostFor is a safety expiry that auto-releases a crashed holder's lock so future triggers are not blocked forever. lockAtLeastFor is a minimum hold that guards against clock skew, preventing a fast job from being picked up again within the same window. Leader election makes one instance the leader for all singleton periodic work, and Quartz clustering uses a shared JDBC job store for persistent triggers with misfire handling.
At National Bank of India, four replicas were each generating every customer's statement, so customers received four copies. Rajesh fixed it with ShedLock, guarding the monthly job with a named lock. Crucially, the job remained idempotent, upserting statements by customer and period, because even run-once mechanisms have edge cases around lock expiry or leadership flips, and idempotency is the ultimate safety net.
Remember one sentence: guard every @Scheduled job in a multi-instance deployment with ShedLock, leader election, or Quartz clustering, and keep the job idempotent as the final safety net.
---