Definition
- Silent load transfer -- the fallback redirects the full primary load onto a dependency not provisioned for it.
- Stale/incorrect data -- returning a default or last-known-good value can be worse than an error for correctness-critical operations.
- Observability masking -- the fallback catches exceptions and returns success, so the primary failure is invisible to dashboards.
- Fix: bound the fallback -- bulkhead plus rate limit so the fallback dependency receives only what it can handle; shed the excess.
- Fix: honest answers -- mark degraded data as degraded; fail closed for debit operations.
- Fix: alert on degradation -- emit a distinct metric every time the fallback fires.
- Test independently -- a fallback you have not load-tested is a hope, not a safety net.
National Bank of India has a fast express counter (Redis) and a slow main counter (PostgreSQL). The manager adds a rule: if express is closed, send everyone to main. But express handled 90% of customers, and main was staffed for 10%. The day express breaks, the crowd floods main, which collapses. Worse, the dashboard still shows 'balance lookups succeeding' because the fallback returns success codes.
Understanding the Concepts
How does a fallback make things worse?
A fallback is intended to preserve availability when the primary path fails. But it runs exactly when the system is already under stress, so a badly designed fallback converts a small failure into a larger one.
package com.bank.resilience;
import java.math.BigDecimal;
public class FallbackDanger {
public static void main(String[] args) {
System.out.println("Normal: Redis serves 90% of balance reads");
System.out.println("Redis goes DOWN");
System.out.println("Fallback: ALL 100% of reads hit PostgreSQL");
System.out.println("PostgreSQL was sized for 10% miss rate");
System.out.println("Result: 10x overload --> PostgreSQL dies --> TOTAL outage");
}
}
Expected output:
Normal: Redis serves 90% of balance reads
Redis goes DOWN
Fallback: ALL 100% of reads hit PostgreSQL
PostgreSQL was sized for 10% miss rate
Result: 10x overload --> PostgreSQL dies --> TOTAL outage
What is the silent load transfer danger?
When Redis serves 90% of balance reads, PostgreSQL is sized for the 10% miss rate. When Redis fails, the fallback redirects all traffic to PostgreSQL. The database receives 10 times its expected load and collapses.
package com.bank.resilience;
import java.math.BigDecimal;
public class SilentLoadTransfer {
// WRONG: unbound fallback transfers full load
public BigDecimal getBalance(String accountId) {
try {
return getFromRedis(accountId); // Redis down!
} catch (Exception e) {
return getFromDb(accountId); // ALL traffic now hits DB
// DB was sized for 10% miss rate --> 10x overload --> DB dies
}
}
private BigDecimal getFromRedis(String id) {
throw new RuntimeException("Redis unavailable");
}
private BigDecimal getFromDb(String id) {
return new BigDecimal("150000.00");
}
}
What is the stale data danger?
Returning a default or zero balance when the primary fails can be worse than an honest error. A debit operation based on a zero balance would deny Rajesh access to his own money.
package com.bank.resilience;
import java.math.BigDecimal;
public class StaleDataDanger {
// WRONG: returning a default balance
public BigDecimal getBalanceWrong(String accountId) {
try {
return getFromRedis(accountId);
} catch (Exception e) {
return BigDecimal.ZERO; // Rajesh told he has no money!
}
}
// CORRECT: for debit operations, fail closed
public BigDecimal getBalanceForDebit(String accountId) {
try {
return getFromRedis(accountId);
} catch (Exception e) {
throw new ServiceUnavailableException(
"Cannot verify balance for " + accountId);
// Honest error is safer than a confident wrong answer
}
}
private BigDecimal getFromRedis(String id) {
throw new RuntimeException("Redis unavailable");
}
public static void main(String[] args) {
StaleDataDanger svc = new StaleDataDanger();
System.out.println("WRONG balance: " + svc.getBalanceWrong("SAV2001"));
try {
svc.getBalanceForDebit("SAV2001");
} catch (Exception e) {
System.out.println("CORRECT: " + e.getMessage());
}
}
}
class ServiceUnavailableException extends RuntimeException {
public ServiceUnavailableException(String msg) { super(msg); }
}
Expected output:
WRONG balance: 0
CORRECT: Cannot verify balance for SAV2001
What is the observability masking danger?
The fallback catches the exception and returns a success response, so the success-rate dashboard stays green. Nobody knows Redis is down until PostgreSQL also collapses.
package com.bank.resilience;
import io.micrometer.core.instrument.MeterRegistry;
import java.math.BigDecimal;
public class ObservabilityMasking {
// WRONG: fallback catches exception, dashboard stays GREEN
// Nobody knows Redis is down until PostgreSQL also dies
// CORRECT: emit distinct fallback metric
public BigDecimal getBalanceWithMetric(String accountId,
MeterRegistry meterRegistry) {
try {
return getFromRedis(accountId);
} catch (Exception e) {
meterRegistry.counter("balance.fallback.activated").increment();
// Alert fires on this counter --> loud signal
return getFromDb(accountId);
}
}
private BigDecimal getFromRedis(String id) {
throw new RuntimeException("Redis unavailable");
}
private BigDecimal getFromDb(String id) {
return new BigDecimal("150000.00");
}
}
How do you fix each danger?
| Problem | Fix |
|---|---|
| Silent load transfer | Bulkhead plus rate limit on fallback; shed excess |
| Stale data | Mark degraded; fail closed for debit operations |
| Observability masking | Distinct fallback-activation metric plus alert |
| Untested fallback | Game-day: kill Redis under load, verify DB survives |
package com.bank.resilience;
import io.github.resilience4j.bulkhead.Bulkhead;
import io.github.resilience4j.bulkhead.BulkheadConfig;
import java.math.BigDecimal;
public class BoundedFallback {
// CORRECT: bulkhead caps fallback at DB's proven capacity
private final Bulkhead fallbackBulkhead = Bulkhead.of("db-fallback",
BulkheadConfig.custom().maxConcurrentCalls(20).build());
public BigDecimal getBalanceSafe(String accountId) {
try {
return getFromRedis(accountId);
} catch (Exception e) {
return Bulkhead.decorateSupplier(fallbackBulkhead,
() -> getFromDb(accountId)).get();
// Excess beyond 20 concurrent --> rejected --> shed load
}
}
private BigDecimal getFromRedis(String id) {
throw new RuntimeException("Redis unavailable");
}
private BigDecimal getFromDb(String id) {
return new BigDecimal("150000.00");
}
}
Why must you test fallback capacity independently?
The fallback path runs only during failure, so it is never exercised under real load during normal operations. It is the least-tested code that runs at exactly the worst moment.
package com.bank.resilience;
public class GameDayTest {
public static void main(String[] args) {
System.out.println("Game-day test plan:");
System.out.println("1. Run load test at production traffic volume");
System.out.println("2. Kill Redis");
System.out.println("3. Verify: PostgreSQL survives (bulkhead sheds excess)");
System.out.println("4. Verify: fallback-activation alert fires");
System.out.println("5. Verify: degraded answers are marked as degraded");
System.out.println("6. Verify: debit operations fail closed, not silently wrong");
}
}
Expected output:
Game-day test plan:
1. Run load test at production traffic volume
2. Kill Redis
3. Verify: PostgreSQL survives (bulkhead sheds excess)
4. Verify: fallback-activation alert fires
5. Verify: degraded answers are marked as degraded
6. Verify: debit operations fail closed, not silently wrong
What are common mistakes?
The most common mistake is transferring the full primary load to the fallback without bounding it.
package com.bank.resilience;
public class CommonMistakes {
public static void main(String[] args) {
// WRONG: transferring full load to fallback without bounding
System.out.println("WRONG: Redis fails --> 100% of reads hit DB");
System.out.println("DB overload --> total outage (worse than Redis alone)");
// CORRECT: bulkhead caps fallback at DB's proven capacity
System.out.println("CORRECT: bulkhead allows max 20 concurrent to DB");
System.out.println("Excess gets 503 --> but DB survives");
}
}
Expected output:
WRONG: Redis fails --> 100% of reads hit DB
DB overload --> total outage (worse than Redis alone)
CORRECT: bulkhead allows max 20 concurrent to DB
Excess gets 503 --> but DB survives
What should a senior engineer know?
- A fallback you have not load-tested independently is a hope, not a safety net.
- Bound the fallback so it receives only what it can handle; reject the rest.
- An honest error is safer than a confident wrong answer for money operations.
- Fallback-activation is a loud signal, not a silent one; alert immediately.
- The fallback path is the least-tested code at the worst possible moment.
package com.bank.resilience;
public class SeniorInsights {
public static void main(String[] args) {
System.out.println("1. Untested fallback = hope, not safety net");
System.out.println("2. Bound with bulkhead; shed excess as 503");
System.out.println("3. Honest error > confident wrong answer for money");
System.out.println("4. Fallback-activation metric = loud alert");
System.out.println("5. Game-day test: kill primary under load, observe");
}
}
Expected output:
1. Untested fallback = hope, not safety net
2. Bound with bulkhead; shed excess as 503
3. Honest error > confident wrong answer for money
4. Fallback-activation metric = loud alert
5. Game-day test: kill primary under load, observe
Visual Overview
Fallback Anti-Pattern
┌──────────┐ ┌──────────┐
│ 1000 req │ ──────► │ Redis │ DOWN
│ /sec │ └──────────┘
└──────────┘ │
fallback ▼
┌──────────┐ ┌──────────┐
│ 1000 req │ ──────► │PostgreSQL│ sized for 100 req/s
│ /sec │ │ OVERLOAD!│ total outage
└──────────┘ └──────────┘
Fixed: Bulkhead on Fallback
┌──────────┐ ┌───────────┐ ┌──────────┐
│ 1000 req │──►│ Bulkhead │──►│PostgreSQL│
│ /sec │ │ max 100 │ │ (safe) │
└──────────┘ │ shed 900 │ └──────────┘
└───────────┘
900 requests get 503
but DB survives
Key Takeaway
A fallback is intended to preserve availability when the primary path fails, but it runs exactly when the system is already under stress, so a badly designed fallback converts a small failure into a larger one. The three dangers are silent load transfer (Redis fails and all traffic overwhelms PostgreSQL), stale or incorrect data (returning a default balance that causes wrong debit decisions), and observability masking (the fallback catches exceptions so dashboards stay green while the primary is down).
The fixes are to bound the fallback with a bulkhead and rate limit so the fallback dependency receives only what it can handle, to mark degraded answers as degraded and fail closed for correctness-critical operations like debits, and to emit a distinct metric on every fallback activation so running degraded is a loud signal.
At National Bank of India, Pankaj capped the DB fallback at 20 concurrent requests with a bulkhead, shed the excess as 503s, and set an alert on the fallback-activation counter. A Redis outage no longer cascaded into a database outage. The most important lesson was that a fallback you have not load-tested independently is a hope, not a safety net.
Remember one sentence: a fallback runs when the system is already in trouble, so bound it with a bulkhead, mark degraded answers honestly, alert on activation, and load-test the fallback path independently.
---