Definition
- Timeout – every remote call has a bounded wait so a hanging dependency never ties up threads indefinitely.
- Retry with backoff – the caller retries a bounded number of times with growing delays and jitter, absorbing transient blips.
- Circuit breaker – once failures cross a threshold the breaker opens and fails instantly, protecting both caller and callee.
- Fallback – when the primary path fails, a degraded but useful response such as a cached value or default is returned.
- Bulkhead – limits concurrent calls per dependency so one slow service cannot exhaust all threads.
- Graceful degradation – the system continues to provide core value with reduced functionality rather than failing completely.
- Monitoring – circuit breaker state, retry counts, and timeout rates emit metrics and alerts to close the loop.
Imagine the National Bank of India's branch depends on a central credit-score office to approve loans. One day the credit office phone line is dead. A graceful teller does not keep Rajesh waiting at the counter forever (that is a timeout), does not redial frantically hundreds of times (that is retry with backoff), and after noticing the line has been dead for a while, stops dialing altogether for a few minutes to save effort (that is a circuit breaker). Instead the teller tells Rajesh "we will give you a provisional decision based on your existing relationship" (that is a fallback and graceful degradation) and quietly files a report so the manager knows the line is down (that is monitoring).
Understanding the Concepts
What happens when a microservice goes down?
When a downstream microservice goes down, the caller must contain the blast radius so it stays responsive and the failure does not cascade through the system. Without protection, the caller's threads pile up waiting on the dead service, its own callers time out, and the whole chain collapses. The goal is graceful degradation: the system continues to provide core value with reduced functionality rather than failing completely.
package com.bank.transfer;
import io.github.resilience4j.bulkhead.annotation.Bulkhead;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import io.github.resilience4j.timelimiter.annotation.TimeLimiter;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class FraudCheckService {
private final FraudServiceClient client;
public FraudCheckService(FraudServiceClient client) {
this.client = client;
}
@TimeLimiter(name = "fraudTl")
@CircuitBreaker(name = "fraudCb", fallbackMethod = "fraudFallback")
@Retry(name = "fraudRetry")
@Bulkhead(name = "fraudBh")
public RiskScore checkFraud(String accountId, BigDecimal amount) {
return client.evaluate(accountId, amount);
}
public RiskScore fraudFallback(String accountId, BigDecimal amount,
Throwable t) {
// Conservative default: flag for manual review
return new RiskScore("MEDIUM", true);
}
}
Normal flow: FraudService returns LOW risk -> transfer proceeds
Service down: Circuit breaker opens -> fallback returns MEDIUM + flagged for review
Why is a timeout the most important defense?
A timeout ensures every remote call has a bounded wait. The worst failure is not an error but a call that hangs forever and ties up a thread. In a Spring Boot application using WebClient, you configure response timeouts directly. For Resilience4j, you use the TimeLimiter annotation.
# application.yml
resilience4j:
timelimiter:
instances:
fraudTl:
timeout-duration: 2s
cancel-running-future: true
package com.bank.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.reactive.function.client.WebClient;
import java.time.Duration;
import io.netty.channel.ChannelOption;
import reactor.netty.http.client.HttpClient;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
@Configuration
public class WebClientConfig {
@Bean
public WebClient fraudWebClient() {
HttpClient httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 500)
.responseTimeout(Duration.ofSeconds(2));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.baseUrl("http://fraud-service")
.build();
}
}
Without timeout: thread blocked 30+ seconds -> pool exhausted -> gateway down
With 2s timeout: thread freed in 2s -> breaker trips -> fallback served
How does retry with exponential backoff work?
Retry with backoff gives transient failures a second chance without creating a thundering herd. If a call fails, you wait a short, growing delay with a little randomness (jitter) and try again a limited number of times. The key rule is to only retry idempotent operations. Retrying a non-idempotent transfer could double-charge Rajesh.
# application.yml
resilience4j:
retry:
instances:
fraudRetry:
max-attempts: 3
wait-duration: 200ms
enable-exponential-backoff: true
exponential-backoff-multiplier: 2
enable-randomized-wait: true
randomized-wait-factor: 0.5
retry-exceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
Retry sequence:
Attempt 1: immediate -> TimeoutException
Attempt 2: wait ~300ms -> TimeoutException
Attempt 3: wait ~600ms -> Success OR circuit breaker trips
What is a circuit breaker and why does it matter?
A circuit breaker stops calling a service that is clearly down. It watches the recent failure rate, and when failures cross a threshold it opens, meaning subsequent calls fail instantly without even attempting the network. This protects both the caller (no wasted threads) and the callee (no pile-on while it recovers). After a cooldown it goes half-open, letting a few trial calls through. If they succeed it closes again; if they fail it reopens.
resilience4j:
circuitbreaker:
instances:
fraudCb:
sliding-window-type: COUNT_BASED
sliding-window-size: 10
failure-rate-threshold: 50
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 3
minimum-number-of-calls: 5
State machine:
CLOSED -- failure rate >= 50% --> OPEN
OPEN -- 30s cooldown --> HALF_OPEN
HALF_OPEN -- trials pass --> CLOSED
HALF_OPEN -- trials fail --> OPEN
What is a bulkhead and why is it needed?
A bulkhead isolates resources so one sick dependency cannot sink the whole ship. Named after watertight compartments in a ship's hull, it limits how many concurrent calls or threads a given dependency may consume. If the statement-generation service becomes slow, a bulkhead caps it at, say, ten concurrent calls, so the remaining threads stay free to serve transfers for Rajesh and Priya.
resilience4j:
bulkhead:
instances:
fraudBh:
max-concurrent-calls: 10
max-wait-duration: 500ms
package com.bank.transfer;
import io.github.resilience4j.bulkhead.annotation.Bulkhead;
import org.springframework.stereotype.Service;
@Service
public class StatementService {
@Bulkhead(name = "statementBh", fallbackMethod = "statementFallback")
public String generateStatement(String accountId) {
// Calls external statement generation
return statementClient.generate(accountId);
}
public String statementFallback(String accountId, Throwable t) {
return "Statement generation is temporarily unavailable. " +
"Please try again in a few minutes.";
}
}
Without bulkhead: slow statement service consumes all 200 threads -> everything slow
With bulkhead: statement service capped at 10 threads -> 190 threads serve transfers
How does a fallback provide graceful degradation?
A fallback is the graceful answer when the primary path fails. It might return a cached value, a sensible default, or a queued-for-later acknowledgment. For example, if the fraud-scoring service is down, the fallback applies a conservative default risk score and flags the transaction for later review, so the transfer is not simply refused.
package com.bank.transfer;
import java.math.BigDecimal;
public class TransferOrchestrator {
private final FraudCheckService fraudCheck;
private final AccountService accountService;
public TransferOrchestrator(FraudCheckService fraudCheck,
AccountService accountService) {
this.fraudCheck = fraudCheck;
this.accountService = accountService;
}
public TransferResult transfer(String from, String to,
BigDecimal amount) {
// Fraud check returns fallback if service is down
RiskScore risk = fraudCheck.checkFraud(from, amount);
if (risk.isHighRisk()) {
return TransferResult.blocked("High risk transfer blocked");
}
// Transfer proceeds even if fraud service was down
// (flagged for manual review via fallback)
accountService.debit(from, amount);
accountService.credit(to, amount);
return TransferResult.success(
risk.isFlaggedForReview()
? "Transfer completed (pending fraud review)"
: "Transfer completed"
);
}
}
Fraud service UP: LOW risk -> transfer completes normally
Fraud service DOWN: MEDIUM risk (fallback) -> transfer completes, flagged for review
Fraud service DOWN: HIGH amount -> blocked even with fallback (conservative)
How does monitoring close the loop?
Monitoring and alerting tie everything together by turning silent fallbacks into visible signals. Circuit breaker state changes, retry counts, timeout rates, and fallback invocations should emit metrics to Prometheus and alerts to the on-call engineer. A silent fallback that masks an outage for days is its own kind of failure.
package com.bank.config;
import io.micrometer.core.instrument.MeterRegistry;
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
@Component
public class CircuitBreakerHealthIndicator implements HealthIndicator {
private final CircuitBreaker fraudCb;
public CircuitBreakerHealthIndicator(CircuitBreaker fraudCb) {
this.fraudCb = fraudCb;
}
@Override
public Health health() {
CircuitBreaker.State state = fraudCb.getState();
if (state == CircuitBreaker.State.OPEN) {
return Health.down()
.withDetail("fraudCircuitBreaker", "OPEN")
.withDetail("failureRate",
fraudCb.getMetrics().getFailureRate())
.build();
}
return Health.up()
.withDetail("fraudCircuitBreaker", state.name())
.build();
}
}
Prometheus metrics exposed:
resilience4j_circuitbreaker_state{name="fraudCb"} = 1 (OPEN)
resilience4j_retry_calls_total{name="fraudRetry",kind="successful_without_retry"}
resilience4j_bulkhead_available_concurrent_calls{name="fraudBh"}
resilience4j_timelimiter_calls_total{name="fraudTl",kind="timeout"}
How is this different from gateway-level handling vs service-level handling?
Gateway-level resilience protects the edge, while service-level resilience protects individual dependencies. Both are needed for defense in depth.
| Aspect | Gateway-Level | Service-Level |
|---|---|---|
| Scope | All inbound traffic | Per-dependency calls |
| Tools | Rate limiting, global timeout | Circuit breaker, retry, bulkhead |
| Fallback | Generic error page | Domain-specific cached data |
| Configuration | Centralized | Per-service tuned |
| Example | Kong, Spring Cloud Gateway | Resilience4j annotations |
Defense in depth:
Client -> [Gateway: rate limit, global timeout]
-> [Service A: timeout + CB + retry + bulkhead]
-> [Service B: may also have its own CB + retry]
Visual Overview
Incoming Request
│
▼
┌──────────────────────────────────────────┐
│ Transfer Service │
│ │
│ ┌────────────────────────────────────┐ │
│ │ TimeLimiter (2s) │ │
│ │ ┌────────────────────────────┐ │ │
│ │ │ CircuitBreaker (50%) │ │ │
│ │ │ ┌──────────────────────┐ │ │ │
│ │ │ │ Retry (3 attempts) │ │ │ │
│ │ │ │ ┌────────────────┐ │ │ │ │
│ │ │ │ │ Bulkhead │ │ │ │ │
│ │ │ │ │ (max 10) │ │ │ │ │ ┌──────────────┐
│ │ │ │ │ ┌──────────┐ │ │ │ │ │ │ Fallback │
│ │ │ │ │ │ Call │ │ │ │────┼───┼───>│ Cached value │
│ │ │ │ │ │ Fraud Svc│ │ │ │ │ │ │ or default │
│ │ │ │ │ └──────────┘ │ │ │ │ │ └──────────────┘
│ │ │ │ └────────────────┘ │ │ │ │
│ │ │ └──────────────────────┘ │ │ │
│ │ └────────────────────────────┘ │ │
│ └────────────────────────────────────┘ │
└──────────────────────────────────────────┘
Key Takeaway
When one microservice is down, the goal is to contain the blast radius so the caller stays responsive and the failure does not cascade. The toolkit is layered: a timeout on every remote call as the first defense, because the worst failure is not an error but a call that hangs forever and ties up threads. On top of timeouts, retry with exponential backoff and jitter gives transient blips a second chance, but only for idempotent operations. Then a circuit breaker watches the failure rate and opens when failures are sustained, so subsequent calls fail instantly without touching the network.
The piece that keeps customers served is the fallback. When the fraud-check service went down at National Bank of India, instead of rejecting every transfer, the fallback applied a conservative default risk score for low-value transfers and flagged them for later manual review, so legitimate customers like Rajesh were still served while high-risk cases stayed protected. Bulkheads cap how many concurrent calls a single dependency can consume, so if statement generation becomes slow it cannot starve the threads that handle transfers.
None of this is complete without monitoring and alerting. Circuit breaker state changes, retry counts, timeout rates, and fallback invocations all emit metrics to Prometheus, and we alert the on-call engineer when the breaker opens or the fallback rate spikes, because a silent fallback that quietly masks an outage for days is its own failure.
Remember one sentence: fail fast with timeouts, retry transient errors with backoff, trip a circuit breaker to stop the bleeding, serve a fallback so the user still gets a useful answer, isolate with bulkheads, and watch everything with metrics and alerts.
---