Definition
- SET NX PX -- atomic set-if-not-exists with expiry; exactly one of the racing instances wins.
- TTL (time to live) -- auto-expires the lock so a crashed holder never deadlocks the system permanently.
- Unique token -- the lock value is a UUID; prevents the wrong-release bug where instance A deletes instance B's lock.
- Lua release -- a server-side script that checks the token and deletes atomically, closing the race between GET and DEL.
- Idempotent job -- the lock makes double execution rare; idempotency makes it harmless if it ever happens.
- Redisson / Redlock -- library-level solutions for lock leasing, watchdog renewal, and multi-node consensus.
National Bank of India runs the nightly interest calculation on five servers for high availability. At 2 AM all five wake up and try to calculate interest for Rajesh and Priya, which would credit interest five times. A single key hangs on a hook in a shared room (Redis). Whoever grabs the key first does the job; everyone else sees the hook is empty and goes back to sleep. The key is tagged with the name of who took it, so only that person can return it. If the holder faints, the key automatically falls back on the hook after a set time.
Understanding the Concepts
What problem does a Redis distributed lock solve?
When you scale a service horizontally for availability, every instance has the same scheduled job. Without coordination, a cron at 2 AM fires on all five instances and the interest posting runs five times. You need mutual exclusion across machines. A local Java lock cannot provide this because each JVM has its own lock object. Redis, being a single shared fast store, becomes the coordination point.
package com.bank.scheduler;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
@Component
public class InterestPostingJob {
// Without a distributed lock, all 5 instances run this at 2 AM
@Scheduled(cron = "0 0 2 * * *")
public void postDailyInterest() {
// Rajesh's SAV2001 gets interest posted 5 times!
System.out.println("Posting interest for all accounts...");
}
}
Expected output (WRONG -- runs on all 5 instances):
Instance-1: Posting interest for all accounts...
Instance-2: Posting interest for all accounts...
Instance-3: Posting interest for all accounts...
Instance-4: Posting interest for all accounts...
Instance-5: Posting interest for all accounts...
How does setIfAbsent with TTL work?
The core operation is Redis SET key value NX PX milliseconds. NX means "only set if the key does not already exist," which is atomic, so exactly one of the racing instances wins. PX sets an expiry in milliseconds. In Spring Data Redis this is opsForValue().setIfAbsent() with a Duration, which returns true to the one winner and false to everyone else.
package com.bank.lock;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import java.time.Duration;
import java.util.UUID;
@Component
public class RedisDistributedLock {
private final StringRedisTemplate redis;
public RedisDistributedLock(StringRedisTemplate redis) {
this.redis = redis;
}
public String tryAcquire(String lockKey, Duration ttl) {
String token = UUID.randomUUID().toString();
Boolean acquired = redis.opsForValue()
.setIfAbsent(lockKey, token, ttl); // SET NX PX
return Boolean.TRUE.equals(acquired) ? token : null;
}
}
Expected output:
Instance-1 calls tryAcquire --> token = "a1b2c3d4" (won)
Instance-2 calls tryAcquire --> token = null (lost)
Instance-3 calls tryAcquire --> token = null (lost)
Why is a TTL mandatory?
If the winner acquires the lock and then its JVM crashes or the machine dies before releasing, without a TTL the lock key stays forever. The job never runs again on any instance -- a permanent deadlock. The TTL is a safety valve: the lock auto-expires, so the next scheduled run can acquire it. Set the TTL comfortably longer than the worst-case job duration so the lock does not expire while the job is still running.
package com.bank.lock;
public class TtlReasoning {
public static void main(String[] args) {
long worstCaseJobMs = 30_000; // 30 seconds
long safetyMargin = 2; // 2x multiplier
long ttlMs = worstCaseJobMs * safetyMargin;
System.out.println("Worst-case job duration: " + worstCaseJobMs + " ms");
System.out.println("TTL with safety margin: " + ttlMs + " ms");
// TTL = 60 seconds; lock auto-expires even if holder crashes
}
}
Expected output:
Worst-case job duration: 30000 ms
TTL with safety margin: 60000 ms
Why must the lock value be a unique token?
The lock value must be a unique identifier (UUID or instance-id plus timestamp). This prevents the classic wrong-release bug: instance A acquires the lock, the job runs long and the TTL expires, Redis frees the lock, instance B acquires it, and then instance A finishes and blindly deletes the lock -- but it just deleted B's lock. By storing a unique token and releasing only if the stored token equals mine, A sees the token no longer matches and does not delete B's lock.
package com.bank.lock;
public class WrongReleaseBug {
// WRONG: release without checking the token
public void releaseUnsafe(String lockKey) {
// redis.delete(lockKey);
// Instance A deletes Instance B's lock!
}
// CORRECT: release only if the token matches (see Lua script below)
public void releaseSafe(String lockKey, String myToken) {
// Lua script checks: if GET(key) == myToken then DEL(key)
System.out.println("Only delete if stored token equals: " + myToken);
}
}
Why must release be atomic with a Lua script?
Releasing naively means GET the key, check if it equals my token, then DEL it. But between the GET and the DEL, the TTL could expire and another instance could acquire. A Lua script runs atomically on the Redis server, so the check and delete happen as one indivisible operation.
package com.bank.lock;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import java.util.Collections;
public class AtomicRelease {
private static final String LUA_RELEASE =
"if redis.call('get', KEYS[1]) == ARGV[1] then " +
" return redis.call('del', KEYS[1]) " +
"else " +
" return 0 " +
"end";
private final StringRedisTemplate redis;
public AtomicRelease(StringRedisTemplate redis) {
this.redis = redis;
}
public boolean release(String lockKey, String token) {
DefaultRedisScript<Long> script = new DefaultRedisScript<>(LUA_RELEASE, Long.class);
Long result = redis.execute(script, Collections.singletonList(lockKey), token);
return result != null && result == 1L;
// Returns 1 if deleted (my lock), 0 if token mismatch (not mine)
}
}
Expected output:
Instance A calls release with token "a1b2c3d4" --> 1 (deleted own lock)
Instance B calls release with token "x9y8z7" on A's lock --> 0 (mismatch, not deleted)
How do you put it all together for a scheduled task?
The complete pattern: acquire with setIfAbsent, run the job if acquired, release with Lua in a finally block.
package com.bank.scheduler;
import com.bank.lock.RedisDistributedLock;
import com.bank.lock.AtomicRelease;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.time.Duration;
@Component
public class InterestPostingJobSafe {
private static final String LOCK_KEY = "lock:interest-posting";
private static final Duration TTL = Duration.ofSeconds(60);
private final RedisDistributedLock lock;
private final AtomicRelease release;
public InterestPostingJobSafe(RedisDistributedLock lock, AtomicRelease release) {
this.lock = lock;
this.release = release;
}
@Scheduled(cron = "0 0 2 * * *")
public void postDailyInterest() {
String token = lock.tryAcquire(LOCK_KEY, TTL);
if (token == null) {
System.out.println("Another instance holds the lock. Skipping.");
return;
}
try {
System.out.println("Lock acquired. Posting interest for SAV2001...");
// ... actual interest calculation for Rajesh, Priya ...
} finally {
release.release(LOCK_KEY, token);
System.out.println("Lock released.");
}
}
}
Expected output (Instance 1 wins):
Instance-1: Lock acquired. Posting interest for SAV2001...
Instance-1: Lock released.
Instance-2: Another instance holds the lock. Skipping.
Instance-3: Another instance holds the lock. Skipping.
How do you handle long-running jobs with a watchdog?
If a job might run longer than the TTL, a background watchdog periodically extends the TTL via a token-checked Lua script that calls PEXPIRE only if the token matches. This is what Redisson does automatically with lock leases.
package com.bank.lock;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class LockWatchdog {
private static final String LUA_EXTEND =
"if redis.call('get', KEYS[1]) == ARGV[1] then " +
" return redis.call('pexpire', KEYS[1], ARGV[2]) " +
"else return 0 end";
private final ScheduledExecutorService scheduler =
Executors.newSingleThreadScheduledExecutor();
public void startWatchdog(String lockKey, String token, long ttlMs) {
long renewInterval = ttlMs / 3; // renew at 1/3 of TTL
scheduler.scheduleAtFixedRate(() -> {
// Extend TTL only if we still own the lock
System.out.println("Watchdog: extending TTL for " + lockKey);
}, renewInterval, renewInterval, TimeUnit.MILLISECONDS);
}
public void stopWatchdog() {
scheduler.shutdown();
}
}
What are common mistakes?
The most frequent mistakes involve missing TTL, missing token check, and not making the job idempotent.
package com.bank.lock;
public class CommonMistakes {
// WRONG 1: no TTL -- holder crash = permanent deadlock
// redis.opsForValue().setIfAbsent("lock:interest", "locked");
// CORRECT 1: always set TTL
// redis.opsForValue().setIfAbsent("lock:interest", token, Duration.ofSeconds(60));
// WRONG 2: constant lock value -- anyone can release
// redis.opsForValue().setIfAbsent("lock:interest", "LOCKED", ttl);
// redis.delete("lock:interest"); // any instance deletes it
// CORRECT 2: unique token + Lua release
// WRONG 3: job is not idempotent -- rare double-run doubles interest
// CORRECT 3: check posting_date marker so double-run is harmless
public static void main(String[] args) {
System.out.println("WRONG: no TTL, constant value, non-idempotent job");
System.out.println("CORRECT: TTL + UUID token + Lua release + idempotent job");
}
}
Expected output:
WRONG: no TTL, constant value, non-idempotent job
CORRECT: TTL + UUID token + Lua release + idempotent job
What should a senior engineer know?
A single-Redis lock is not a perfect distributed lock under network partitions and Redis failover. It is a practical "mostly once" guard. For stronger guarantees, use Redisson, which implements the Redlock algorithm, lock leasing, watchdog renewal, and reentrancy. For money-critical jobs, always combine the lock with an idempotent job so that even a rare double-run does not double-post.
- The lock makes double execution rare; idempotency makes it harmless.
- Redisson's watchdog automatically extends the lock while the job runs.
- For critical financial operations, add a posting-date marker so the job is truly idempotent.
- Be honest about guarantees: a single-Redis lock is practical, not perfect.
package com.bank.scheduler;
import java.time.LocalDate;
public class IdempotentInterestPosting {
// Before posting, check if today's interest was already posted
public boolean alreadyPosted(String accountId, LocalDate date) {
// SELECT count(*) FROM interest_postings
// WHERE account_id = ? AND posting_date = ?
return false; // stub
}
public void post(String accountId, LocalDate date) {
if (alreadyPosted(accountId, date)) {
System.out.println("Already posted for " + accountId + " on " + date);
return;
}
System.out.println("Posting interest for " + accountId + " on " + date);
// INSERT INTO interest_postings ...
}
}
Expected output (double-run scenario):
First run: Posting interest for SAV2001 on 2024-01-15
Second run: Already posted for SAV2001 on 2024-01-15
Visual Overview
Redis Distributed Lock for Scheduled Task
2 AM: all 5 instances wake up
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│Instance-1│ │Instance-2│ │Instance-3│ │Instance-4│ │Instance-5│
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │ │ │
└────────────┴────────────┼────────────┴────────────┘
▼
┌─────────────────────────┐
│ Redis: SET NX PX │
│ key = lock:interest │
│ value = UUID token │
│ TTL = 60s │
└─────────────────────────┘
│
Instance-1 wins (true)
Others get false --> skip
│
┌─────────────────────────┐
│ Instance-1 runs job │
│ Posts interest for │
│ Rajesh (SAV2001) │
└────────────┬────────────┘
│
┌─────────────────────────┐
│ Lua: if GET == myToken │
│ then DEL │
│ Lock released atomically│
└─────────────────────────┘
Key Takeaway
A Redis distributed lock guarantees that a scheduled job runs once per schedule rather than once per instance. At National Bank of India the nightly interest posting ran on five instances, and without coordination all five credited Rajesh and Priya, quintupling their interest. The fix uses the atomic SET NX PX operation, exposed in Spring as setIfAbsent with a TTL, so exactly one instance wins and the others skip.
Two subtle correctness points matter. First, the lock value must be a unique token (UUID), not a constant, because of the wrong-release bug: if instance A's job outlasts the TTL, Redis frees the lock, instance B acquires it, and then A finishes and blindly deletes B's lock. Storing a unique token and releasing only when the token matches prevents this. Second, the release must be atomic via a Lua script that reads the value and deletes it only if it equals the token in a single indivisible server-side operation.
A single-Redis lock is a practical "mostly once" guard, not a perfect distributed lock under Redis failover. For money-critical jobs like interest posting, Pankaj combined the lock with an idempotent job that checks a posting-date marker, so even a rare double-run never doubles the interest credit.
Remember one sentence: acquire with setIfAbsent using NX plus a TTL so exactly one instance wins and a crash cannot deadlock, store a unique token and release with an atomic Lua script, and back the lock with an idempotent job for true run-once safety.
---