Definition
- Circuit breaker – A stability mechanism that tracks recent failure rate and, once it crosses a threshold, opens to reject calls instantly instead of hanging.
- Three states – CLOSED (calls pass, outcomes recorded), OPEN (calls rejected with CallNotPermittedException), HALF_OPEN (trial calls test recovery).
- Sliding window – Count-based (last N calls) or time-based (last N seconds) for computing the failure rate.
- Retry – A separate pattern that re-attempts a failed call a bounded number of times with backoff; for transient glitches.
- Aspect-order gotcha – By default Retry wraps CircuitBreaker, so Retry sees the open circuit's exception as another failure and retries pointlessly.
Imagine National Bank of India's app calls an external credit-score service. If that service is down, calling it again and again wastes time and ties up threads. A circuit breaker is like a smart fuse: after too many failures it trips and immediately says "do not even try right now", so customers get a fast fallback. After a cooling-off period it cautiously lets one call through to test recovery. The gotcha is that if the retry logic is outside the fuse, it keeps flipping the fuse switch even after it has tripped.
Understanding the Concepts
Why does a circuit breaker exist?
When a downstream service is slow or down, naive callers keep sending requests that each wait for a timeout, consuming threads until the caller itself fails. This is how one failure cascades into a system-wide outage.
package com.bank.resilience;
public class CascadeWithoutBreaker {
public static void main(String[] args) {
System.out.println("Without circuit breaker:");
System.out.println(" Credit-score service is down");
System.out.println(" Each call waits for 30s timeout");
System.out.println(" 200 threads all waiting -> thread pool exhausted");
System.out.println(" Loan service cannot serve ANY request -> cascade failure");
System.out.println();
System.out.println("With circuit breaker:");
System.out.println(" After 10 failures, circuit opens");
System.out.println(" Calls fail fast in 1ms with fallback");
System.out.println(" Threads freed instantly -> loan service keeps working");
}
}
Expected output:
Without circuit breaker:
Credit-score service is down
Each call waits for 30s timeout
200 threads all waiting -> thread pool exhausted
Loan service cannot serve ANY request -> cascade failure
With circuit breaker:
After 10 failures, circuit opens
Calls fail fast in 1ms with fallback
Threads freed instantly -> loan service keeps working
How does Resilience4j implement the three states?
CLOSED: calls pass through, outcomes recorded in a sliding window. OPEN: calls rejected immediately. HALF_OPEN: a few trial calls test if the dependency recovered.
package com.bank.resilience;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.stereotype.Service;
@Service
public class CreditScoreClient {
@CircuitBreaker(name = "creditScore", fallbackMethod = "fallbackScore")
public int getCreditScore(String accountId) {
// When failure rate > 50%, circuit opens
// Calls are rejected with CallNotPermittedException
return externalCreditService.getScore(accountId);
}
public int fallbackScore(String accountId, Throwable t) {
System.out.println("Circuit breaker active for " + accountId
+ ", reason: " + t.getClass().getSimpleName());
return 650; // Safe default score
}
}
resilience4j:
circuitbreaker:
instances:
creditScore:
slidingWindowType: COUNT_BASED
slidingWindowSize: 10
failureRateThreshold: 50
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 3
slowCallRateThreshold: 80
slowCallDurationThreshold: 3s
State transitions:
CLOSED: failure rate 30% (< 50%) -> stays CLOSED
CLOSED: failure rate 60% (> 50%) -> transitions to OPEN
OPEN: 30 seconds pass -> transitions to HALF_OPEN
HALF_OPEN: 3 trial calls succeed -> transitions to CLOSED
HALF_OPEN: 2 of 3 trial calls fail -> transitions back to OPEN
How does Retry differ from Circuit Breaker?
Retry re-attempts a failed call. It is for transient glitches. Circuit Breaker stops calling entirely when a dependency is clearly down.
| Aspect | Retry | Circuit Breaker |
|---|---|---|
| Purpose | Handle transient failures | Prevent cascade from sustained failure |
| Behavior | Re-attempt the same call | Stop making calls entirely |
| Duration | Seconds (backoff between attempts) | Minutes (wait in open state) |
| Best for | Network blip, brief timeout | Dependency genuinely down |
| Risk if used alone | Amplifies load on a dying service | Does not recover from transient blips |
package com.bank.resilience;
import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.stereotype.Service;
@Service
public class PaymentClient {
@Retry(name = "payment", fallbackMethod = "fallbackPayment")
public String processPayment(String accountId) {
// Retry: tries 3 times with exponential backoff
// Good for: network blip, momentary 503
// Bad for: payment service genuinely down for 10 minutes
return paymentService.charge(accountId);
}
public String fallbackPayment(String accountId, Throwable t) {
return "Payment queued for retry";
}
}
resilience4j:
retry:
instances:
payment:
maxAttempts: 3
waitDuration: 1s
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
retryExceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
Retry behavior:
Attempt 1: fails (IOException)
Wait 1s
Attempt 2: fails (IOException)
Wait 2s
Attempt 3: succeeds -> return result
OR Attempt 3: fails -> fallback
What is the default aspect-order gotcha?
By default, Retry is outermost wrapping CircuitBreaker. When the circuit is OPEN, the breaker throws CallNotPermittedException. Retry sees this as just another failure and retries, defeating the purpose of failing fast.
package com.bank.resilience;
public class AspectOrderGotcha {
// DEFAULT ORDER: Retry -> CircuitBreaker -> actual call
//
// Circuit is OPEN:
// Attempt 1: CircuitBreaker throws CallNotPermittedException
// Retry sees failure -> waits 1s
// Attempt 2: CircuitBreaker throws CallNotPermittedException (still open!)
// Retry sees failure -> waits 2s
// Attempt 3: CircuitBreaker throws CallNotPermittedException
// -> 3 wasted retries, 3 seconds lost, OPEN circuit was already saying "stop"
// CORRECT ORDER: CircuitBreaker -> Retry -> actual call
// Circuit is OPEN:
// CircuitBreaker throws CallNotPermittedException immediately
// NO retries, fails fast in 1ms
// Circuit is CLOSED:
// Call goes through Retry -> actual call
// If transient failure, Retry handles it inside the closed circuit
public static void main(String[] args) {
System.out.println("DEFAULT (WRONG): Retry wraps CircuitBreaker");
System.out.println(" Open circuit -> Retry retries CallNotPermittedException 3x");
System.out.println(" Wastes 3 seconds, defeats fail-fast purpose");
System.out.println();
System.out.println("CORRECT: CircuitBreaker wraps Retry");
System.out.println(" Open circuit -> fails fast immediately, 0 retries");
System.out.println(" Closed circuit -> Retry handles transient failures inside");
}
}
Expected output:
DEFAULT (WRONG): Retry wraps CircuitBreaker
Open circuit -> Retry retries CallNotPermittedException 3x
Wastes 3 seconds, defeats fail-fast purpose
CORRECT: CircuitBreaker wraps Retry
Open circuit -> fails fast immediately, 0 retries
Closed circuit -> Retry handles transient failures inside
How do you fix the aspect order?
Configure the aspect order so CircuitBreaker is outermost, or tell Retry to ignore CallNotPermittedException.
# Fix 1: Reorder aspects so CircuitBreaker is outermost
resilience4j:
circuitbreaker:
circuitBreakerAspectOrder: 1
retry:
retryAspectOrder: 2
# Lower number = outermost. CircuitBreaker (1) wraps Retry (2)
# Fix 2: Tell Retry to ignore CallNotPermittedException
resilience4j:
retry:
instances:
payment:
ignoreExceptions:
- io.github.resilience4j.circuitbreaker.CallNotPermittedException
Fix 1 behavior:
Open circuit -> CircuitBreaker rejects immediately -> Retry never sees it
Fix 2 behavior:
Open circuit -> Retry sees CallNotPermittedException -> skips retry -> fallback
Why must you pair a circuit breaker with a timeout?
A circuit breaker cannot trip on calls that never return. Without a timeout, a hanging call holds a thread forever and the breaker never sees a failure to count.
package com.bank.resilience;
public class TimeoutPairing {
public static void main(String[] args) {
System.out.println("WRONG: CircuitBreaker without timeout");
System.out.println(" Call hangs for 5 minutes -> thread blocked");
System.out.println(" Breaker sees no failure (call hasn't returned)");
System.out.println(" Breaker stays CLOSED -> more threads hang -> cascade");
System.out.println();
System.out.println("CORRECT: Timeout + CircuitBreaker");
System.out.println(" Call exceeds 3s timeout -> TimeoutException thrown");
System.out.println(" Breaker counts it as a failure");
System.out.println(" After threshold -> circuit opens -> calls fail fast");
}
}
Expected output:
WRONG: CircuitBreaker without timeout
Call hangs for 5 minutes -> thread blocked
Breaker sees no failure (call hasn't returned)
Breaker stays CLOSED -> more threads hang -> cascade
CORRECT: Timeout + CircuitBreaker
Call exceeds 3s timeout -> TimeoutException thrown
Breaker counts it as a failure
After threshold -> circuit opens -> calls fail fast
How do you use it with a fallback?
The fallback returns a safe default when the circuit is open or the call fails.
package com.bank.resilience;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class InterestRateClient {
@CircuitBreaker(name = "interestRate", fallbackMethod = "defaultRate")
public BigDecimal getCurrentRate(String productId) {
return externalRateService.getRate(productId);
}
public BigDecimal defaultRate(String productId, Throwable t) {
// Return last known rate from cache, not a guess
System.out.println("Using cached rate for " + productId);
return new BigDecimal("7.50");
}
}
Expected output (circuit open):
Using cached rate for HOME-LOAN-30Y
What should a senior engineer know?
Expose the circuit state as a metric (Prometheus gauge), size the window and thresholds to traffic volume, combine with a bulkhead, and use slowCallRateThreshold to catch unhealthy-but-up dependencies.
Senior checklist:
[x] Circuit state exposed as Prometheus metric
[x] Aspect order: CircuitBreaker outermost
[x] Timeout paired with every circuit breaker
[x] Fallback is genuinely safe (cached value, not another risky call)
[x] slowCallRateThreshold catches slow-but-up dependencies
[x] Bulkhead paired to isolate thread pools
[x] Window and thresholds tuned to traffic volume
Visual Overview
Circuit Breaker State Machine:
┌─────────────────────────────────────────┐
│ CLOSED │
│ Calls pass through normally │
│ Outcomes recorded in sliding window │
│ Failure rate: 30% │
└──────────────┬──────────────────────────┘
│ failure rate > 50%
▼
┌─────────────────────────────────────────┐
│ OPEN │
│ All calls REJECTED immediately │
│ CallNotPermittedException thrown │
│ Wait duration: 30 seconds │
└──────────────┬──────────────────────────┘
│ wait duration expires
▼
┌─────────────────────────────────────────┐
│ HALF_OPEN │
│ 3 trial calls permitted │
│ If succeed -> back to CLOSED │
│ If fail -> back to OPEN │
└─────────────────────────────────────────┘
Aspect Order:
WRONG: Retry ──▶ CircuitBreaker ──▶ Call
(retries open circuit's exception)
CORRECT: CircuitBreaker ──▶ Retry ──▶ Call
(open circuit fails fast, retry only inside closed)
Key Takeaway
The Circuit Breaker pattern stops a service from repeatedly calling a dependency that is failing. When a downstream is slow or down, naive callers keep sending requests that each wait for a timeout, consuming threads until the caller itself fails. The breaker watches outcomes and, once failures are frequent enough, opens so calls fail fast with a fallback, giving the dependency room to recover.
Resilience4j implements this as a three-state machine. CLOSED: calls pass through, outcomes recorded in a sliding window. When the failure rate crosses the threshold, it transitions to OPEN, where every call is rejected with CallNotPermittedException. After a wait duration, it moves to HALF_OPEN to test recovery. Retry is a complementary pattern for transient glitches, not sustained failure. The two combine: Retry absorbs brief blips, the breaker protects against prolonged outages.
The gotcha Pankaj learned at National Bank of India is the default aspect order: Retry wraps CircuitBreaker, so when the circuit is open, Retry sees the CallNotPermittedException and wastefully retries against an open circuit. The fix is to reorder so CircuitBreaker wraps Retry, which means once the circuit opens, the whole call fails fast with zero wasted retries. Also always pair the breaker with a timeout, because a breaker cannot trip on calls that never return.
Remember one sentence: a circuit breaker trips open after too many failures so calls fail fast, Retry merely re-attempts transient failures, and you must reorder their aspects so the breaker is outermost or the retry wastes time against an open circuit.
---