Back to Data Management & Consistency
Q6Question 6 of 12

How do distributed locks work with Redis in microservices

Answer: A distributed lock is a coordination mechanism that ensures only one process across multiple service instances can execute a critical section at a time, typically implemented using Redis SET NX with an automatic expiry to prevent deadlocks.

Key Concepts
  • A distributed lock ensures that only one process across multiple service instances can execute a critical section at a time.
  • Redis SET with NX (set if not exists) and PX (expiry in milliseconds) is the foundation of Redis-based distributed locking.
  • Redisson provides a production-ready distributed lock with automatic lease renewal (watchdog), reentrancy, and fair ordering.
  • Lock lease time prevents deadlocks: if the lock holder crashes, the lock automatically expires after the lease duration.
  • Fencing tokens are monotonically increasing numbers attached to each lock acquisition that prevent stale lock holders from executing after their lock expires.
  • The Redlock algorithm uses multiple independent Redis masters for higher reliability but adds complexity and latency.
  • Distributed locks are a last resort. Prefer idempotent operations or database-level locking when possible.

Definition

A distributed lock is a coordination mechanism that ensures only one process across multiple service instances can execute a critical section at a time, typically implemented using Redis SET NX with an automatic expiry to prevent deadlocks.
  • Distributed lock is a mutual exclusion mechanism that works across multiple JVM processes, servers, or containers, unlike Java's synchronized keyword which only works within a single JVM.
  • SET NX (Not eXists) is a Redis command that sets a key only if it does not already exist, providing an atomic "acquire lock" operation.
  • Lease time is the duration after which the lock automatically expires. This prevents deadlocks when the lock holder crashes.
  • Fencing token is a monotonically increasing number issued with each lock acquisition. The protected resource rejects operations from tokens that are lower than the highest token it has seen.
  • Redisson is a Java Redis client that provides production-ready distributed locks with automatic lease renewal (watchdog), reentrancy, and fair ordering.

Think of the single key to the bank vault at National Bank of India. Only one employee can hold the key at a time. If the key holder gets sick and goes home, the key automatically returns to the front desk after one hour (lease time). The key has a serial number (fencing token) stamped on it, and the vault only opens if the serial number is higher than the last one used. This prevents an employee who took too long and lost the key from accidentally opening the vault after someone else already used it with a newer key.

Understanding the Concepts

What is a distributed lock?

A distributed lock ensures that only one process across multiple service instances can execute a critical section at the same time. In a single JVM, you use synchronized or ReentrantLock. In a distributed system with multiple service instances, those do not work because each instance has its own memory.

A distributed lock uses an external coordination service (Redis, ZooKeeper, or a database) that all instances can access.

JAVA
package com.bank.service;

import org.redisson.api.RLock;
import org.redisson.api.RedissonClient;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.concurrent.TimeUnit;
import java.util.logging.Logger;

@Service
public class PaymentService {

    private static final Logger log =
        Logger.getLogger(PaymentService.class.getName());

    private final RedissonClient redisson;
    private final AccountRepository repository;

    public PaymentService(RedissonClient redisson,
            AccountRepository repository) {
        this.redisson = redisson;
        this.repository = repository;
    }

    public void processScheduledPayment(String paymentId,
            String fromAccount, String toAccount,
            BigDecimal amount) {

        // Lock on the payment ID to prevent duplicate processing
        RLock lock = redisson.getLock("lock:payment:" + paymentId);

        boolean acquired = false;
        try {
            // Wait up to 5 seconds to acquire, auto-release after 30s
            acquired = lock.tryLock(5, 30, TimeUnit.SECONDS);
            if (!acquired) {
                log.warning("Could not acquire lock for payment: "
                    + paymentId);
                return;
            }

            // Critical section: only one instance processes this payment
            log.info("Processing payment " + paymentId);
            executeTransfer(fromAccount, toAccount, amount);
            markPaymentComplete(paymentId);

        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            if (acquired && lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }

    private void executeTransfer(String from, String to,
            BigDecimal amount) {
        repository.debit(from, amount);
        repository.credit(to, amount);
    }

    private void markPaymentComplete(String paymentId) {
        // Update payment status in database
    }
}
TEXT
// Scenario: 3 instances pick up the same scheduled payment
// Instance A: acquires lock for payment:PAY-001, processes it
// Instance B: tryLock returns false, skips (duplicate prevented)
// Instance C: tryLock returns false, skips (duplicate prevented)
// Result: payment PAY-001 is processed exactly once.

Why do you need distributed locks in banking?

Without a distributed lock, multiple service instances can process the same payment simultaneously.

ScenarioWithout LockWith Lock
Scheduled EMI paymentDebited twice from SAV2001Debited once
Concurrent balance updateLost update (one write overwrites the other)Serialized updates
Rate limit checkTwo threads both pass the limitOnly one passes
Batch job processingSame batch processed by two podsExactly-once processing
JAVA
// Problem: Without distributed lock
// Instance A reads balance: 75000
// Instance B reads balance: 75000
// Instance A deducts 5000, writes 70000
// Instance B deducts 5000, writes 70000 (WRONG! should be 65000)
// Lost update: one deduction is lost
TEXT
// Solution: With distributed lock
// Instance A acquires lock, reads 75000, deducts 5000, writes 70000, releases lock
// Instance B acquires lock, reads 70000, deducts 5000, writes 65000, releases lock
// Correct: both deductions applied

How does the basic Redis lock work?

The foundation of a Redis lock is the SET key value NX PX milliseconds command. NX means "set only if the key does not exist." PX sets an expiry in milliseconds.

JAVA
package com.bank.lock;

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import java.time.Duration;
import java.util.UUID;

@Component
public class SimpleRedisLock {

    private final StringRedisTemplate redis;

    public SimpleRedisLock(StringRedisTemplate redis) {
        this.redis = redis;
    }

    /**
     * Try to acquire a lock.
     * Returns a unique token if successful, null if not.
     */
    public String tryAcquire(String lockKey, Duration leaseTime) {
        String token = UUID.randomUUID().toString();
        Boolean acquired = redis.opsForValue()
            .setIfAbsent(lockKey, token, leaseTime);
        return Boolean.TRUE.equals(acquired) ? token : null;
    }

    /**
     * Release the lock, but only if we still own it.
     * Uses a Lua script for atomicity.
     */
    public boolean release(String lockKey, String token) {
        String script =
            "if redis.call('get', KEYS[1]) == ARGV[1] then " +
            "  return redis.call('del', KEYS[1]) " +
            "else " +
            "  return 0 " +
            "end";
        Long result = redis.execute(
            new org.springframework.data.redis.core.script
                .DefaultRedisScript<>(script, Long.class),
            java.util.List.of(lockKey), token);
        return result != null && result == 1;
    }
}
TEXT
Redis commands:
SET lock:payment:PAY-001 "uuid-abc-123" NX PX 30000
-> OK (lock acquired)

SET lock:payment:PAY-001 "uuid-def-456" NX PX 30000
-> nil (lock not acquired, already held)

// Release with Lua script:
GET lock:payment:PAY-001 -> "uuid-abc-123"
// Matches our token, so DEL succeeds
DEL lock:payment:PAY-001

The Lua script for release is critical. Without it, a race condition can occur: Thread A checks that it owns the lock, then Thread A's lease expires, Thread B acquires the lock, and then Thread A deletes Thread B's lock. The Lua script makes the check-and-delete atomic.

What is Redisson and why use it over a basic lock?

Redisson is a Java Redis client with production-ready distributed primitives. Its RLock provides features that a basic SET NX lock does not.

FeatureBasic SET NX LockRedisson RLock
Automatic lease renewalNoYes (watchdog)
ReentrancyNoYes
Fair orderingNoYes (RFairLock)
ReadWrite lockNoYes (RReadWriteLock)
Lock with timeoutManualBuilt-in tryLock
Pub/sub notification on releaseNoYes (waiters notified)
JAVA
package com.bank.config;

import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.config.Config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RedissonConfig {

    @Bean
    public RedissonClient redissonClient() {
        Config config = new Config();
        config.useSingleServer()
            .setAddress("redis://localhost:6379")
            .setConnectionPoolSize(20)
            .setConnectionMinimumIdleSize(5);
        return Redisson.create(config);
    }
}
TEXT
// Redisson watchdog:
// If you call lock.lock() without a lease time,
// Redisson starts a watchdog timer that automatically
// extends the lock every 10 seconds (default: 30s/3).
// This prevents the lock from expiring while the
// critical section is still running.

// If the JVM crashes, the watchdog stops, and the lock
// expires after the lease time. No deadlock.

What is a fencing token and why does it matter?

A fencing token is a monotonically increasing number issued with each lock acquisition. It prevents a stale lock holder from executing after their lock has expired.

  • Scenario without fencing:
  • Client A acquires lock with lease 30s.
  • Client A pauses (GC, network delay) for 35 seconds.
  • Lock expires. Client B acquires the lock.
  • Client A resumes and writes to the database, thinking it still holds the lock.
  • Both A and B wrote. Data is corrupted.
  • With fencing:
  • Client A acquires lock, gets fencing token 42.
  • Client A pauses for 35 seconds.
  • Lock expires. Client B acquires lock, gets fencing token 43.
  • Client B writes with token 43.
  • Client A resumes, tries to write with token 42.
  • The database rejects token 42 because it already accepted 43.
JAVA
package com.bank.service;

import org.redisson.api.RLock;
import org.redisson.api.RedissonClient;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.concurrent.TimeUnit;

@Service
public class FencedPaymentService {

    private final RedissonClient redisson;
    private final StringRedisTemplate redis;
    private final AccountRepository repository;

    public FencedPaymentService(RedissonClient redisson,
            StringRedisTemplate redis,
            AccountRepository repository) {
        this.redisson = redisson;
        this.redis = redis;
        this.repository = repository;
    }

    public void processPayment(String paymentId,
            String fromAccount, BigDecimal amount) {

        RLock lock = redisson.getLock("lock:payment:" + paymentId);
        try {
            if (!lock.tryLock(5, 30, TimeUnit.SECONDS)) {
                return;
            }

            // Generate fencing token
            Long fencingToken = redis.opsForValue()
                .increment("fence:payment:" + paymentId);

            // Pass fencing token to the database
            int updated = repository.debitWithFence(
                fromAccount, amount, fencingToken);

            if (updated == 0) {
                // Fencing token was stale. Another process won.
                throw new StaleTokenException(
                    "Fencing token " + fencingToken + " is stale");
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            if (lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}
TEXT
// Database-side fencing:
UPDATE accounts
SET balance = balance - 5000.00,
    last_fence_token = 43
WHERE account_id = 'SAV2001'
  AND (last_fence_token IS NULL OR last_fence_token < 43);

// If last_fence_token is already 43 or higher, 0 rows updated.
// The stale process with token 42 is rejected.

What are common mistakes with distributed locks?

Mistake 1: Not setting a lease time

WRONG: Acquiring a lock without expiry. If the process crashes, the lock is held forever.

JAVA
// WRONG: No lease time. If the process crashes,
// no one can ever acquire this lock again.
redis.opsForValue().setIfAbsent("lock:payment:PAY-001", "owner-1");
// DEADLOCK: lock persists forever if owner-1 crashes

CORRECT: Always set a lease time.

JAVA
// CORRECT: Lock expires after 30 seconds even if owner crashes.
redis.opsForValue().setIfAbsent("lock:payment:PAY-001",
    "owner-1", Duration.ofSeconds(30));

Mistake 2: Deleting without ownership check

WRONG: Deleting the lock key without checking who owns it.

JAVA
// WRONG: Thread A's lock expires. Thread B acquires the lock.
// Thread A deletes the lock key. Now Thread B's lock is gone.
redis.delete("lock:payment:PAY-001");

CORRECT: Use a Lua script to check ownership before deleting.

JAVA
// CORRECT: Only delete if the token matches (see SimpleRedisLock above).
// The Lua script makes check-and-delete atomic.
release("lock:payment:PAY-001", myToken);

What should a senior or principal engineer know?

- Redlock algorithm. Martin Kleppmann critiqued Redis single-instance locks because Redis replication is asynchronous. If the master fails before replicating the lock to a replica, the lock can be acquired twice. Redlock uses N independent Redis masters (typically 5) and requires acquiring the lock on a majority (3 of 5). This provides higher reliability at the cost of complexity and latency.

- Distributed locks vs database locks. For simple cases, a database advisory lock (PostgreSQL pg_advisory_lock) or a SELECT ... FOR UPDATE row lock may be simpler and more reliable than a Redis lock. Use Redis locks when the protected resource is not the database itself (for example, external API calls or file operations).

- Idempotent operations vs locks. The best lock is no lock. If you can make the operation idempotent (store a unique payment ID in the database with a unique constraint), you do not need a distributed lock at all. The database constraint prevents duplicates.

- Lock contention monitoring. Monitor how long threads wait to acquire locks and how often tryLock fails. High contention indicates a design problem. Consider breaking the work into smaller, independent units.

- Spring Integration distributed lock. Spring Integration provides a LockRegistry abstraction with Redis, JDBC, and ZooKeeper implementations. This gives a Spring-native API for distributed locking.

SQL
-- Idempotent alternative to distributed lock:
-- Use a unique constraint on payment_id
CREATE TABLE processed_payments (
    payment_id  VARCHAR(50) PRIMARY KEY,
    processed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- Before processing, INSERT with ON CONFLICT DO NOTHING
INSERT INTO processed_payments (payment_id)
VALUES ('PAY-001')
ON CONFLICT (payment_id) DO NOTHING;
-- If 0 rows inserted, the payment was already processed.
-- No lock needed.
TEXT
// Instance A: INSERT PAY-001 -> 1 row inserted -> process payment
// Instance B: INSERT PAY-001 -> 0 rows inserted -> skip (already processed)
// No distributed lock needed. The database handles uniqueness.

What are common interview follow-up questions?

  • What is the difference between a distributed lock and a database row lock?
  • What happens if the Redis master fails while a lock is held?
  • What is the Redlock algorithm and when do you use it?
  • How does Redisson's watchdog prevent lock expiry during long operations?
  • What is a fencing token and why is it important?
  • When should you use idempotent operations instead of distributed locks?
  • How do you monitor lock contention in production?
  • What is the difference between a fair lock and an unfair lock?

Visual Overview

TEXT
  Distributed Lock Flow
  =====================

  Instance A             Redis            Instance B
  +---------+     +---------------+     +---------+
  | tryLock | --> | SET NX PX     | <-- | tryLock |
  |         |     | lock:PAY-001  |     |         |
  |         |     | = "token-A"   |     |         |
  +---------+     +---------------+     +---------+
       |                                     |
   acquired                             NOT acquired
       |                                     |
       v                                     v
  Process PAY-001                         Skip (already
       |                                  being processed)
       v
  unlock (Lua: check token, delete)
       |
       v
  Lock released
TEXT
  Fencing Token Flow
  ==================

  Client A: lock acquired, fence=42
       |                                    
  (GC pause, 35 seconds...)                 
       |                                    
  Lock expires (30s lease)
       |                                    
  Client B: lock acquired, fence=43
  Client B: writes with fence=43 -> OK
       |                                    
  Client A resumes: writes with fence=42
  Database: fence 42 < 43 -> REJECTED
       |                                    
  Data integrity preserved.

Key Takeaway

Distributed locks solve a fundamental problem in microservices: ensuring that only one instance processes a critical operation at a time. When three instances of the payment service pick up the same scheduled EMI payment, a distributed lock on the payment ID ensures only one instance processes it. The others see that the lock is held and skip.

Redis provides the building blocks with SET NX PX: set a key only if it does not exist, with an automatic expiry. Redisson wraps this in a production-ready API with automatic lease renewal (watchdog), reentrancy, fair ordering, and pub/sub-based notifications when a lock is released.

The lease time is critical. Without it, a crashed process holds the lock forever, causing a deadlock. With a lease of 30 seconds, the lock automatically expires and another process can pick it up. Fencing tokens add an extra safety layer: even if a stale process resumes after its lock expired, the database rejects its write because a newer fencing token was already accepted.

Distributed locks are a last resort. If you can make the operation idempotent using a database unique constraint, that is simpler and more reliable. Use distributed locks for non-idempotent operations, external API calls, or coordinating access to resources that the database does not control.

Remember one sentence: use Redis SET NX PX for the lock, always set a lease time to prevent deadlocks, check ownership before unlocking, and prefer idempotent operations when possible.

---

Share
Use ← → arrow keys to navigate