Back to Caching: Spring Cache & Redis
Q5Question 5 of 5

What happens when Redis fails and all traffic hits the database

Answer: When Redis fails, every request becomes a cache miss, all traffic hits the database directly, connection pools exhaust, and the database can collapse under the sudden load unless you have circuit breakers, stale fallbacks, and request coalescing in place.

Key Concepts
  • When Redis fails, every cache miss triggers a database query, causing a sudden spike in database load called a thundering herd.
  • A circuit breaker on the cache layer detects Redis failures and fails fast, preventing threads from blocking on connection timeouts.
  • Request coalescing (or request collapsing) ensures that when multiple threads request the same key, only one thread queries the database and shares the result.
  • Stale fallback serves the last known cached value from a local fallback store when Redis is unavailable, accepting temporary staleness over failure.
  • Gradual cache warming after Redis recovery avoids a second thundering herd by slowly repopulating the cache instead of letting all requests hit the database at once.
  • Connection pool exhaustion is the immediate symptom of Redis failure because all threads block waiting for a Redis connection that will never come.
  • Design your application so that cache failure degrades performance but never causes a complete outage.

Definition

When Redis fails, every request becomes a cache miss, all traffic hits the database directly, connection pools exhaust, and the database can collapse under the sudden load unless you have circuit breakers, stale fallbacks, and request coalescing in place.
  • Thundering herd is the scenario where a cache failure causes all concurrent requests to simultaneously hit the database, overwhelming it.
  • Circuit breaker is a resilience pattern that monitors failures and, after a threshold, short-circuits calls to the failing service, returning a fallback response immediately.
  • Request coalescing (also called request collapsing or singleflight) deduplicates concurrent requests for the same key so only one thread queries the database.
  • Stale fallback serves the last known value from a local store (Caffeine, HashMap) when Redis is unavailable, trading freshness for availability.
  • Gradual warming repopulates the cache slowly after recovery, using random TTL jitter and rate-limited pre-loading to avoid a second thundering herd.

Think of the information desk at National Bank of India's main branch. The desk has a printed rate card (Redis cache) that customers check for exchange rates. If the rate card is lost (Redis failure), every customer walks to the back office (database) to ask individually. Ten customers asking at once overwhelms the back office. A circuit breaker is a sign on the desk that says "Rate card temporarily unavailable, use yesterday's rates" (stale fallback). Request coalescing is a teller saying "I am already asking the back office. Wait here and I will share the answer with all of you."

Understanding the Concepts

What happens when Redis fails?

When Redis becomes unavailable (network partition, out of memory, crash, or maintenance), every cache read returns a timeout or connection error. The application falls back to the database for every request.

The impact cascades:

  1. Connection pool exhaustion. Each thread waits for a Redis connection that will never come. The default Lettuce timeout is 60 seconds. Within seconds, all threads in the pool are blocked.
  2. Database overload. All requests that were served from cache now hit the database. If the cache was absorbing 95% of reads, the database suddenly receives 20x its normal load.
  3. Database connection pool exhaustion. The database cannot handle the spike. Connection requests queue up. Queries time out.
  4. Cascade failure. Users see errors. Retries make it worse. The entire system becomes unresponsive.
JAVA
package com.bank.service;

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;
import java.util.logging.Logger;

@Service
public class NaiveBalanceService {

    private static final Logger log =
        Logger.getLogger(NaiveBalanceService.class.getName());

    private final StringRedisTemplate redis;
    private final AccountRepository repository;

    public NaiveBalanceService(StringRedisTemplate redis,
            AccountRepository repository) {
        this.redis = redis;
        this.repository = repository;
    }

    // PROBLEM: No resilience. Redis failure blocks threads.
    public BigDecimal getBalance(String accountId) {
        try {
            String cached = redis.opsForValue()
                .get("balance:" + accountId);
            if (cached != null) {
                return new BigDecimal(cached);
            }
        } catch (Exception e) {
            // Redis timeout: thread blocked for 60 seconds
            log.severe("Redis error: " + e.getMessage());
        }

        // Every request hits the database
        return repository.findBalance(accountId);
    }
}
TEXT
// Normal operation:
// 1000 requests/sec, 950 served from Redis, 50 hit DB

// Redis failure:
// 1000 requests/sec, ALL hit DB
// DB connection pool: 50 connections
// Queue depth: 950 waiting
// Response time: 50ms -> 5000ms -> timeout
// Users see: HTTP 503 Service Unavailable

How do you add a circuit breaker on the cache layer?

A circuit breaker monitors Redis calls. After a threshold of failures, it "opens" and stops calling Redis entirely, returning a fallback immediately. This prevents threads from blocking on Redis timeouts.

Use Resilience4j, the standard resilience library for Spring Boot.

JAVA
package com.bank.service;

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;
import java.util.logging.Logger;

@Service
public class ResilientBalanceService {

    private static final Logger log =
        Logger.getLogger(ResilientBalanceService.class.getName());

    private final StringRedisTemplate redis;
    private final AccountRepository repository;

    public ResilientBalanceService(StringRedisTemplate redis,
            AccountRepository repository) {
        this.redis = redis;
        this.repository = repository;
    }

    @CircuitBreaker(name = "redisCache",
        fallbackMethod = "getBalanceFromDb")
    public BigDecimal getBalance(String accountId) {
        String cached = redis.opsForValue()
            .get("balance:" + accountId);
        if (cached != null) {
            return new BigDecimal(cached);
        }

        // Cache miss: query DB and populate cache
        BigDecimal balance = repository.findBalance(accountId);
        redis.opsForValue().set("balance:" + accountId,
            balance.toPlainString(), Duration.ofSeconds(60));
        return balance;
    }

    // Fallback: skip Redis entirely
    private BigDecimal getBalanceFromDb(String accountId,
            Throwable t) {
        log.warning("Redis circuit open, falling back to DB: "
            + t.getMessage());
        return repository.findBalance(accountId);
    }
}
TEXT
# application.yml: Resilience4j circuit breaker config
resilience4j:
  circuitbreaker:
    instances:
      redisCache:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
        minimumNumberOfCalls: 5
TEXT
// Flow:
// 1. Redis fails. 5 out of 10 calls fail (50% failure rate).
// 2. Circuit breaker opens. All subsequent calls go to fallback.
// 3. No thread blocks on Redis. Fallback queries DB directly.
// 4. After 30 seconds, circuit enters half-open state.
// 5. 5 test calls go to Redis. If Redis is back, circuit closes.
// 6. Normal caching resumes.

How does request coalescing prevent database overload?

When Redis is down and 100 threads request the balance for SAV2001 at the same time, all 100 threads hit the database. Request coalescing ensures only one thread queries the database. The other 99 threads wait for the result and share it.

JAVA
package com.bank.service;

import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CompletableFuture;
import java.util.logging.Logger;

@Service
public class CoalescingBalanceService {

    private static final Logger log =
        Logger.getLogger(CoalescingBalanceService.class.getName());

    private final AccountRepository repository;

    // In-flight requests: key -> future
    private final ConcurrentHashMap<String,
        CompletableFuture<BigDecimal>> inFlight =
            new ConcurrentHashMap<>();

    public CoalescingBalanceService(AccountRepository repository) {
        this.repository = repository;
    }

    public BigDecimal getBalance(String accountId) {
        CompletableFuture<BigDecimal> future = inFlight
            .computeIfAbsent(accountId, key -> {
                // Only the first thread creates the future
                return CompletableFuture.supplyAsync(() -> {
                    try {
                        return repository.findBalance(key);
                    } finally {
                        inFlight.remove(key);
                    }
                });
            });

        return future.join(); // All threads share the same result
    }
}
TEXT
// Without coalescing:
// 100 concurrent requests for SAV2001
// -> 100 database queries
// DB load: 100x normal

// With coalescing:
// 100 concurrent requests for SAV2001
// -> 1 database query, 99 threads wait and share the result
// DB load: 1x normal

What is stale fallback?

Stale fallback uses a local in-memory cache (Caffeine) as a backup. When Redis is down, the application serves the last known value from the local cache. The data may be slightly stale, but the system stays available.

JAVA
package com.bank.service;

import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;

@Service
public class StaleFallbackBalanceService {

    private final StringRedisTemplate redis;
    private final AccountRepository repository;

    // Local cache: last-resort fallback
    private final Cache<String, BigDecimal> localCache =
        Caffeine.newBuilder()
            .maximumSize(10_000)
            .expireAfterWrite(Duration.ofMinutes(5))
            .build();

    public StaleFallbackBalanceService(StringRedisTemplate redis,
            AccountRepository repository) {
        this.redis = redis;
        this.repository = repository;
    }

    @CircuitBreaker(name = "redisCache",
        fallbackMethod = "getBalanceFallback")
    public BigDecimal getBalance(String accountId) {
        String cached = redis.opsForValue()
            .get("balance:" + accountId);
        if (cached != null) {
            BigDecimal balance = new BigDecimal(cached);
            localCache.put(accountId, balance); // update local cache
            return balance;
        }

        // Cache miss
        BigDecimal balance = repository.findBalance(accountId);
        redis.opsForValue().set("balance:" + accountId,
            balance.toPlainString(), Duration.ofSeconds(60));
        localCache.put(accountId, balance);
        return balance;
    }

    // Fallback when Redis circuit is open
    private BigDecimal getBalanceFallback(String accountId,
            Throwable t) {
        // Try local cache first (stale data)
        BigDecimal stale = localCache.getIfPresent(accountId);
        if (stale != null) {
            return stale; // Stale but available
        }

        // No stale data available, must query database
        BigDecimal balance = repository.findBalance(accountId);
        localCache.put(accountId, balance);
        return balance;
    }
}
TEXT
// Normal operation:
// Redis hit -> return value, also store in localCache

// Redis failure:
// Circuit breaker opens -> fallback method
// localCache has SAV2001 -> return stale value (maybe 60s old)
// localCache does NOT have SAV2003 -> query DB, store in localCache

// Result: most requests served from local cache.
// Only new/unseen keys hit the database.
// Database load stays manageable.

How does gradual cache warming work?

When Redis recovers from a failure, the cache is empty. If all requests immediately try to populate the cache, you get a second thundering herd. Gradual warming avoids this.

  • Techniques:
  • TTL jitter. Add a random offset to each key's TTL so keys do not all expire at the same time.
  • Rate-limited pre-loading. On startup, load the top N most-accessed keys into Redis at a controlled rate.
  • Lazy warming with request coalescing. Let normal traffic populate the cache, but use coalescing to prevent duplicate database queries.
JAVA
package com.bank.service;

import org.springframework.boot.CommandLineRunner;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
import java.util.logging.Logger;

@Component
public class CacheWarmer implements CommandLineRunner {

    private static final Logger log =
        Logger.getLogger(CacheWarmer.class.getName());

    private final StringRedisTemplate redis;
    private final AccountRepository repository;

    public CacheWarmer(StringRedisTemplate redis,
            AccountRepository repository) {
        this.redis = redis;
        this.repository = repository;
    }

    @Override
    public void run(String... args) throws Exception {
        log.info("Starting cache warm-up...");

        // Load the 1000 most frequently accessed accounts
        List<String> topAccounts = repository
            .findTopAccessedAccountIds(1000);

        int loaded = 0;
        for (String accountId : topAccounts) {
            BigDecimal balance = repository.findBalance(accountId);

            // TTL jitter: 45-75 seconds instead of fixed 60
            int jitter = ThreadLocalRandom.current().nextInt(45, 76);
            redis.opsForValue().set(
                "balance:" + accountId,
                balance.toPlainString(),
                Duration.ofSeconds(jitter));

            loaded++;
            if (loaded % 100 == 0) {
                // Rate limit: pause every 100 keys
                Thread.sleep(50);
                log.info("Warmed " + loaded + " keys");
            }
        }

        log.info("Cache warm-up complete: " + loaded + " keys loaded");
    }
}
TEXT
Starting cache warm-up...
Warmed 100 keys
Warmed 200 keys
...
Warmed 1000 keys
Cache warm-up complete: 1000 keys loaded
// Total time: ~500ms (1000 keys / 100 per batch * 50ms pause)
// TTLs range from 45s to 75s, so keys expire at different times.

What are common mistakes when handling Redis failures?

Mistake 1: No timeout on Redis connections

WRONG: Default Lettuce timeout is 60 seconds. Threads block for a full minute.

TEXT
# WRONG: No custom timeout. Default is 60 seconds.
spring.data.redis.host=localhost
spring.data.redis.port=6379
# Thread blocks for 60 seconds before timeout.

CORRECT: Set aggressive timeouts.

TEXT
# CORRECT: Fail fast with 2-second timeout.
spring.data.redis.host=localhost
spring.data.redis.port=6379
spring.data.redis.timeout=2000ms
spring.data.redis.lettuce.pool.max-active=20
spring.data.redis.lettuce.pool.max-wait=500ms

Mistake 2: Treating cache failure as application failure

WRONG: Letting a Redis exception propagate as HTTP 500.

JAVA
// WRONG: Redis exception crashes the request
public BigDecimal getBalance(String accountId) {
    String cached = redis.opsForValue()
        .get("balance:" + accountId);
    // If Redis is down, RedisConnectionFailureException
    // propagates to the controller as HTTP 500.
    return new BigDecimal(cached);
}

CORRECT: Catch the exception, fall back to database.

JAVA
// CORRECT: Cache failure degrades performance, not availability
public BigDecimal getBalance(String accountId) {
    try {
        String cached = redis.opsForValue()
            .get("balance:" + accountId);
        if (cached != null) {
            return new BigDecimal(cached);
        }
    } catch (Exception e) {
        // Log and fall back. Do not crash.
    }
    return repository.findBalance(accountId);
}

What about Redis Sentinel and Cluster for high availability?

Redis Sentinel monitors Redis instances and performs automatic failover. Redis Cluster distributes data across multiple shards.

FeatureRedis SentinelRedis Cluster
PurposeHigh availability (automatic failover)High availability + horizontal scaling
Data distributionSingle master, replicasSharded across N masters
FailoverAutomatic (Sentinel promotes replica)Automatic (cluster promotes replica)
Max data sizeLimited by single node memorySum of all shard memory
ComplexityModerateHigher
TEXT
# application.yml: Redis Sentinel configuration
spring:
  data:
    redis:
      sentinel:
        master: mymaster
        nodes:
          - sentinel1:26379
          - sentinel2:26379
          - sentinel3:26379
      timeout: 2000ms
TEXT
// Sentinel failover flow:
// 1. Master goes down
// 2. Sentinels detect failure after quorum
// 3. Sentinel promotes a replica to master
// 4. Lettuce client receives notification
// 5. Client reconnects to the new master
// 6. ~5 seconds of downtime during failover
// 7. Your circuit breaker covers this gap

What should a senior or principal engineer know?

- Multi-layer caching. Use a local Caffeine cache (L1) in front of Redis (L2). The L1 cache absorbs read spikes without network calls. When Redis fails, L1 continues serving stale data. This is called a near cache.

- Chaos engineering. Regularly test Redis failures in staging. Use Toxiproxy or tc (traffic control) to simulate latency, packet loss, and full outage. Verify that the circuit breaker opens, the fallback activates, and the database handles the extra load.

  • Graceful degradation tiers. Define what happens at each failure level:
  • Level 1: Redis slow (>100ms). Serve from local cache if available.
  • Level 2: Redis down. Circuit breaker open. All reads go to DB.
  • Level 3: DB connection pool near capacity. Enable rate limiting.
  • Level 4: DB overloaded. Return 503 with Retry-After header.

- Redis persistence. Configure RDB snapshots or AOF (Append Only File) so Redis can recover its data after a restart. Without persistence, a Redis restart means an empty cache and a cold-start thundering herd.

- Observability. Monitor these metrics: Redis connection pool utilization, cache hit/miss ratio, circuit breaker state (open/closed/half-open), database connection pool utilization, and P99 response time. Alert when cache hit rate drops below 80%.

JAVA
package com.bank.config;

import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.math.BigDecimal;
import java.time.Duration;

@Configuration
public class MultiLayerCacheConfig {

    @Bean
    public Cache<String, BigDecimal> localBalanceCache() {
        return Caffeine.newBuilder()
            .maximumSize(50_000)             // L1: 50K entries
            .expireAfterWrite(Duration.ofSeconds(30)) // short TTL
            .recordStats()                   // enable metrics
            .build();
    }
}
TEXT
// Multi-layer cache read flow:
// 1. Check L1 (Caffeine, in-memory) -> hit? return
// 2. Check L2 (Redis, network)      -> hit? store in L1, return
// 3. Query database                 -> store in L2 and L1, return
//
// Redis failure:
// 1. Check L1 -> hit? return (stale but fast)
// 2. L2 circuit open -> skip Redis
// 3. Query database -> store in L1, return
// Result: L1 absorbs most reads. DB sees only L1 misses.

What are common interview follow-up questions?

  • What is the thundering herd problem and how do you prevent it?
  • How does a circuit breaker work and when does it open?
  • What is the difference between failing fast and failing gracefully?
  • How do you warm a cache after a Redis restart?
  • What is TTL jitter and why do you use it?
  • How do you monitor Redis health from your Spring Boot application?
  • What is the difference between Redis Sentinel and Redis Cluster?
  • How do you test cache failure scenarios in staging?

Visual Overview

TEXT
  Redis Failure Cascade
  =====================

  Normal:  Client -> Redis (95% hit) -> DB (5%)

  Redis Down:
  Client -> Redis TIMEOUT (2s)
       |
       v
  ALL traffic hits DB
       |
       v
  DB connection pool exhausted
       |
       v
  Queries queue and timeout
       |
       v
  HTTP 503 for users
TEXT
  Resilient Design
  ================

  Client
    |
    v
  L1 Cache (Caffeine, in-memory)
    |-- HIT -> return immediately
    |-- MISS
    v
  Circuit Breaker
    |-- CLOSED -> try Redis (L2)
    |     |-- HIT -> store in L1, return
    |     |-- MISS -> query DB, store in L1+L2, return
    |     |-- ERROR -> increment failure count
    |
    |-- OPEN -> skip Redis
          |-- L1 has stale value? -> return stale
          |-- No stale value -> query DB (with coalescing)

Key Takeaway

Redis failure is not a question of if, but when. A network partition, out-of-memory event, or maintenance window can take Redis down at any time. When that happens, every cache miss becomes a database query, and the database receives 20x its normal load. Without protection, the database connection pool exhausts, queries time out, and users see errors.

The first defense is a circuit breaker on the cache layer. When Redis fails, the circuit breaker opens within seconds and routes all requests directly to the fallback method. This prevents threads from blocking on Redis timeouts. The second defense is stale fallback: a local Caffeine cache stores the last known values and serves them when Redis is unavailable. The data may be 30-60 seconds stale, but the system stays responsive.

Request coalescing ensures that when 100 threads request the same account balance simultaneously, only one thread queries the database. The other 99 wait and share the result. This reduces database load from 100 queries to 1. Gradual warming with TTL jitter prevents a second thundering herd when Redis recovers, because keys expire at different times instead of all at once.

The design principle is simple: cache failure should degrade performance, never cause a complete outage. Set aggressive Redis timeouts (2 seconds, not 60), use a multi-layer cache (L1 local, L2 Redis), monitor cache hit rates, and test failure scenarios with chaos engineering in staging.

Remember one sentence: design so that Redis failure slows the system down, not shuts it down.

---

Share
Use ← → arrow keys to navigate