Definition
- Thundering herd is the scenario where a cache failure causes all concurrent requests to simultaneously hit the database, overwhelming it.
- Circuit breaker is a resilience pattern that monitors failures and, after a threshold, short-circuits calls to the failing service, returning a fallback response immediately.
- Request coalescing (also called request collapsing or singleflight) deduplicates concurrent requests for the same key so only one thread queries the database.
- Stale fallback serves the last known value from a local store (Caffeine, HashMap) when Redis is unavailable, trading freshness for availability.
- Gradual warming repopulates the cache slowly after recovery, using random TTL jitter and rate-limited pre-loading to avoid a second thundering herd.
Think of the information desk at National Bank of India's main branch. The desk has a printed rate card (Redis cache) that customers check for exchange rates. If the rate card is lost (Redis failure), every customer walks to the back office (database) to ask individually. Ten customers asking at once overwhelms the back office. A circuit breaker is a sign on the desk that says "Rate card temporarily unavailable, use yesterday's rates" (stale fallback). Request coalescing is a teller saying "I am already asking the back office. Wait here and I will share the answer with all of you."
Understanding the Concepts
What happens when Redis fails?
When Redis becomes unavailable (network partition, out of memory, crash, or maintenance), every cache read returns a timeout or connection error. The application falls back to the database for every request.
The impact cascades:
- Connection pool exhaustion. Each thread waits for a Redis connection that will never come. The default Lettuce timeout is 60 seconds. Within seconds, all threads in the pool are blocked.
- Database overload. All requests that were served from cache now hit the database. If the cache was absorbing 95% of reads, the database suddenly receives 20x its normal load.
- Database connection pool exhaustion. The database cannot handle the spike. Connection requests queue up. Queries time out.
- Cascade failure. Users see errors. Retries make it worse. The entire system becomes unresponsive.
package com.bank.service;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;
import java.util.logging.Logger;
@Service
public class NaiveBalanceService {
private static final Logger log =
Logger.getLogger(NaiveBalanceService.class.getName());
private final StringRedisTemplate redis;
private final AccountRepository repository;
public NaiveBalanceService(StringRedisTemplate redis,
AccountRepository repository) {
this.redis = redis;
this.repository = repository;
}
// PROBLEM: No resilience. Redis failure blocks threads.
public BigDecimal getBalance(String accountId) {
try {
String cached = redis.opsForValue()
.get("balance:" + accountId);
if (cached != null) {
return new BigDecimal(cached);
}
} catch (Exception e) {
// Redis timeout: thread blocked for 60 seconds
log.severe("Redis error: " + e.getMessage());
}
// Every request hits the database
return repository.findBalance(accountId);
}
}
// Normal operation:
// 1000 requests/sec, 950 served from Redis, 50 hit DB
// Redis failure:
// 1000 requests/sec, ALL hit DB
// DB connection pool: 50 connections
// Queue depth: 950 waiting
// Response time: 50ms -> 5000ms -> timeout
// Users see: HTTP 503 Service Unavailable
How do you add a circuit breaker on the cache layer?
A circuit breaker monitors Redis calls. After a threshold of failures, it "opens" and stops calling Redis entirely, returning a fallback immediately. This prevents threads from blocking on Redis timeouts.
Use Resilience4j, the standard resilience library for Spring Boot.
package com.bank.service;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;
import java.util.logging.Logger;
@Service
public class ResilientBalanceService {
private static final Logger log =
Logger.getLogger(ResilientBalanceService.class.getName());
private final StringRedisTemplate redis;
private final AccountRepository repository;
public ResilientBalanceService(StringRedisTemplate redis,
AccountRepository repository) {
this.redis = redis;
this.repository = repository;
}
@CircuitBreaker(name = "redisCache",
fallbackMethod = "getBalanceFromDb")
public BigDecimal getBalance(String accountId) {
String cached = redis.opsForValue()
.get("balance:" + accountId);
if (cached != null) {
return new BigDecimal(cached);
}
// Cache miss: query DB and populate cache
BigDecimal balance = repository.findBalance(accountId);
redis.opsForValue().set("balance:" + accountId,
balance.toPlainString(), Duration.ofSeconds(60));
return balance;
}
// Fallback: skip Redis entirely
private BigDecimal getBalanceFromDb(String accountId,
Throwable t) {
log.warning("Redis circuit open, falling back to DB: "
+ t.getMessage());
return repository.findBalance(accountId);
}
}
# application.yml: Resilience4j circuit breaker config
resilience4j:
circuitbreaker:
instances:
redisCache:
slidingWindowSize: 10
failureRateThreshold: 50
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 5
minimumNumberOfCalls: 5
// Flow:
// 1. Redis fails. 5 out of 10 calls fail (50% failure rate).
// 2. Circuit breaker opens. All subsequent calls go to fallback.
// 3. No thread blocks on Redis. Fallback queries DB directly.
// 4. After 30 seconds, circuit enters half-open state.
// 5. 5 test calls go to Redis. If Redis is back, circuit closes.
// 6. Normal caching resumes.
How does request coalescing prevent database overload?
When Redis is down and 100 threads request the balance for SAV2001 at the same time, all 100 threads hit the database. Request coalescing ensures only one thread queries the database. The other 99 threads wait for the result and share it.
package com.bank.service;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CompletableFuture;
import java.util.logging.Logger;
@Service
public class CoalescingBalanceService {
private static final Logger log =
Logger.getLogger(CoalescingBalanceService.class.getName());
private final AccountRepository repository;
// In-flight requests: key -> future
private final ConcurrentHashMap<String,
CompletableFuture<BigDecimal>> inFlight =
new ConcurrentHashMap<>();
public CoalescingBalanceService(AccountRepository repository) {
this.repository = repository;
}
public BigDecimal getBalance(String accountId) {
CompletableFuture<BigDecimal> future = inFlight
.computeIfAbsent(accountId, key -> {
// Only the first thread creates the future
return CompletableFuture.supplyAsync(() -> {
try {
return repository.findBalance(key);
} finally {
inFlight.remove(key);
}
});
});
return future.join(); // All threads share the same result
}
}
// Without coalescing:
// 100 concurrent requests for SAV2001
// -> 100 database queries
// DB load: 100x normal
// With coalescing:
// 100 concurrent requests for SAV2001
// -> 1 database query, 99 threads wait and share the result
// DB load: 1x normal
What is stale fallback?
Stale fallback uses a local in-memory cache (Caffeine) as a backup. When Redis is down, the application serves the last known value from the local cache. The data may be slightly stale, but the system stays available.
package com.bank.service;
import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;
@Service
public class StaleFallbackBalanceService {
private final StringRedisTemplate redis;
private final AccountRepository repository;
// Local cache: last-resort fallback
private final Cache<String, BigDecimal> localCache =
Caffeine.newBuilder()
.maximumSize(10_000)
.expireAfterWrite(Duration.ofMinutes(5))
.build();
public StaleFallbackBalanceService(StringRedisTemplate redis,
AccountRepository repository) {
this.redis = redis;
this.repository = repository;
}
@CircuitBreaker(name = "redisCache",
fallbackMethod = "getBalanceFallback")
public BigDecimal getBalance(String accountId) {
String cached = redis.opsForValue()
.get("balance:" + accountId);
if (cached != null) {
BigDecimal balance = new BigDecimal(cached);
localCache.put(accountId, balance); // update local cache
return balance;
}
// Cache miss
BigDecimal balance = repository.findBalance(accountId);
redis.opsForValue().set("balance:" + accountId,
balance.toPlainString(), Duration.ofSeconds(60));
localCache.put(accountId, balance);
return balance;
}
// Fallback when Redis circuit is open
private BigDecimal getBalanceFallback(String accountId,
Throwable t) {
// Try local cache first (stale data)
BigDecimal stale = localCache.getIfPresent(accountId);
if (stale != null) {
return stale; // Stale but available
}
// No stale data available, must query database
BigDecimal balance = repository.findBalance(accountId);
localCache.put(accountId, balance);
return balance;
}
}
// Normal operation:
// Redis hit -> return value, also store in localCache
// Redis failure:
// Circuit breaker opens -> fallback method
// localCache has SAV2001 -> return stale value (maybe 60s old)
// localCache does NOT have SAV2003 -> query DB, store in localCache
// Result: most requests served from local cache.
// Only new/unseen keys hit the database.
// Database load stays manageable.
How does gradual cache warming work?
When Redis recovers from a failure, the cache is empty. If all requests immediately try to populate the cache, you get a second thundering herd. Gradual warming avoids this.
- Techniques:
- TTL jitter. Add a random offset to each key's TTL so keys do not all expire at the same time.
- Rate-limited pre-loading. On startup, load the top N most-accessed keys into Redis at a controlled rate.
- Lazy warming with request coalescing. Let normal traffic populate the cache, but use coalescing to prevent duplicate database queries.
package com.bank.service;
import org.springframework.boot.CommandLineRunner;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
import java.util.logging.Logger;
@Component
public class CacheWarmer implements CommandLineRunner {
private static final Logger log =
Logger.getLogger(CacheWarmer.class.getName());
private final StringRedisTemplate redis;
private final AccountRepository repository;
public CacheWarmer(StringRedisTemplate redis,
AccountRepository repository) {
this.redis = redis;
this.repository = repository;
}
@Override
public void run(String... args) throws Exception {
log.info("Starting cache warm-up...");
// Load the 1000 most frequently accessed accounts
List<String> topAccounts = repository
.findTopAccessedAccountIds(1000);
int loaded = 0;
for (String accountId : topAccounts) {
BigDecimal balance = repository.findBalance(accountId);
// TTL jitter: 45-75 seconds instead of fixed 60
int jitter = ThreadLocalRandom.current().nextInt(45, 76);
redis.opsForValue().set(
"balance:" + accountId,
balance.toPlainString(),
Duration.ofSeconds(jitter));
loaded++;
if (loaded % 100 == 0) {
// Rate limit: pause every 100 keys
Thread.sleep(50);
log.info("Warmed " + loaded + " keys");
}
}
log.info("Cache warm-up complete: " + loaded + " keys loaded");
}
}
Starting cache warm-up...
Warmed 100 keys
Warmed 200 keys
...
Warmed 1000 keys
Cache warm-up complete: 1000 keys loaded
// Total time: ~500ms (1000 keys / 100 per batch * 50ms pause)
// TTLs range from 45s to 75s, so keys expire at different times.
What are common mistakes when handling Redis failures?
Mistake 1: No timeout on Redis connections
WRONG: Default Lettuce timeout is 60 seconds. Threads block for a full minute.
# WRONG: No custom timeout. Default is 60 seconds.
spring.data.redis.host=localhost
spring.data.redis.port=6379
# Thread blocks for 60 seconds before timeout.
CORRECT: Set aggressive timeouts.
# CORRECT: Fail fast with 2-second timeout.
spring.data.redis.host=localhost
spring.data.redis.port=6379
spring.data.redis.timeout=2000ms
spring.data.redis.lettuce.pool.max-active=20
spring.data.redis.lettuce.pool.max-wait=500ms
Mistake 2: Treating cache failure as application failure
WRONG: Letting a Redis exception propagate as HTTP 500.
// WRONG: Redis exception crashes the request
public BigDecimal getBalance(String accountId) {
String cached = redis.opsForValue()
.get("balance:" + accountId);
// If Redis is down, RedisConnectionFailureException
// propagates to the controller as HTTP 500.
return new BigDecimal(cached);
}
CORRECT: Catch the exception, fall back to database.
// CORRECT: Cache failure degrades performance, not availability
public BigDecimal getBalance(String accountId) {
try {
String cached = redis.opsForValue()
.get("balance:" + accountId);
if (cached != null) {
return new BigDecimal(cached);
}
} catch (Exception e) {
// Log and fall back. Do not crash.
}
return repository.findBalance(accountId);
}
What about Redis Sentinel and Cluster for high availability?
Redis Sentinel monitors Redis instances and performs automatic failover. Redis Cluster distributes data across multiple shards.
| Feature | Redis Sentinel | Redis Cluster |
|---|---|---|
| Purpose | High availability (automatic failover) | High availability + horizontal scaling |
| Data distribution | Single master, replicas | Sharded across N masters |
| Failover | Automatic (Sentinel promotes replica) | Automatic (cluster promotes replica) |
| Max data size | Limited by single node memory | Sum of all shard memory |
| Complexity | Moderate | Higher |
# application.yml: Redis Sentinel configuration
spring:
data:
redis:
sentinel:
master: mymaster
nodes:
- sentinel1:26379
- sentinel2:26379
- sentinel3:26379
timeout: 2000ms
// Sentinel failover flow:
// 1. Master goes down
// 2. Sentinels detect failure after quorum
// 3. Sentinel promotes a replica to master
// 4. Lettuce client receives notification
// 5. Client reconnects to the new master
// 6. ~5 seconds of downtime during failover
// 7. Your circuit breaker covers this gap
What should a senior or principal engineer know?
- Multi-layer caching. Use a local Caffeine cache (L1) in front of Redis (L2). The L1 cache absorbs read spikes without network calls. When Redis fails, L1 continues serving stale data. This is called a near cache.
- Chaos engineering. Regularly test Redis failures in staging. Use Toxiproxy or tc (traffic control) to simulate latency, packet loss, and full outage. Verify that the circuit breaker opens, the fallback activates, and the database handles the extra load.
- Graceful degradation tiers. Define what happens at each failure level:
- Level 1: Redis slow (>100ms). Serve from local cache if available.
- Level 2: Redis down. Circuit breaker open. All reads go to DB.
- Level 3: DB connection pool near capacity. Enable rate limiting.
- Level 4: DB overloaded. Return 503 with Retry-After header.
- Redis persistence. Configure RDB snapshots or AOF (Append Only File) so Redis can recover its data after a restart. Without persistence, a Redis restart means an empty cache and a cold-start thundering herd.
- Observability. Monitor these metrics: Redis connection pool utilization, cache hit/miss ratio, circuit breaker state (open/closed/half-open), database connection pool utilization, and P99 response time. Alert when cache hit rate drops below 80%.
package com.bank.config;
import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.math.BigDecimal;
import java.time.Duration;
@Configuration
public class MultiLayerCacheConfig {
@Bean
public Cache<String, BigDecimal> localBalanceCache() {
return Caffeine.newBuilder()
.maximumSize(50_000) // L1: 50K entries
.expireAfterWrite(Duration.ofSeconds(30)) // short TTL
.recordStats() // enable metrics
.build();
}
}
// Multi-layer cache read flow:
// 1. Check L1 (Caffeine, in-memory) -> hit? return
// 2. Check L2 (Redis, network) -> hit? store in L1, return
// 3. Query database -> store in L2 and L1, return
//
// Redis failure:
// 1. Check L1 -> hit? return (stale but fast)
// 2. L2 circuit open -> skip Redis
// 3. Query database -> store in L1, return
// Result: L1 absorbs most reads. DB sees only L1 misses.
What are common interview follow-up questions?
- What is the thundering herd problem and how do you prevent it?
- How does a circuit breaker work and when does it open?
- What is the difference between failing fast and failing gracefully?
- How do you warm a cache after a Redis restart?
- What is TTL jitter and why do you use it?
- How do you monitor Redis health from your Spring Boot application?
- What is the difference between Redis Sentinel and Redis Cluster?
- How do you test cache failure scenarios in staging?
Visual Overview
Redis Failure Cascade
=====================
Normal: Client -> Redis (95% hit) -> DB (5%)
Redis Down:
Client -> Redis TIMEOUT (2s)
|
v
ALL traffic hits DB
|
v
DB connection pool exhausted
|
v
Queries queue and timeout
|
v
HTTP 503 for users
Resilient Design
================
Client
|
v
L1 Cache (Caffeine, in-memory)
|-- HIT -> return immediately
|-- MISS
v
Circuit Breaker
|-- CLOSED -> try Redis (L2)
| |-- HIT -> store in L1, return
| |-- MISS -> query DB, store in L1+L2, return
| |-- ERROR -> increment failure count
|
|-- OPEN -> skip Redis
|-- L1 has stale value? -> return stale
|-- No stale value -> query DB (with coalescing)
Key Takeaway
Redis failure is not a question of if, but when. A network partition, out-of-memory event, or maintenance window can take Redis down at any time. When that happens, every cache miss becomes a database query, and the database receives 20x its normal load. Without protection, the database connection pool exhausts, queries time out, and users see errors.
The first defense is a circuit breaker on the cache layer. When Redis fails, the circuit breaker opens within seconds and routes all requests directly to the fallback method. This prevents threads from blocking on Redis timeouts. The second defense is stale fallback: a local Caffeine cache stores the last known values and serves them when Redis is unavailable. The data may be 30-60 seconds stale, but the system stays responsive.
Request coalescing ensures that when 100 threads request the same account balance simultaneously, only one thread queries the database. The other 99 wait and share the result. This reduces database load from 100 queries to 1. Gradual warming with TTL jitter prevents a second thundering herd when Redis recovers, because keys expire at different times instead of all at once.
The design principle is simple: cache failure should degrade performance, never cause a complete outage. Set aggressive Redis timeouts (2 seconds, not 60), use a multi-layer cache (L1 local, L2 Redis), monitor cache hit rates, and test failure scenarios with chaos engineering in staging.
Remember one sentence: design so that Redis failure slows the system down, not shuts it down.
---