Definition
- Service discovery – the gateway asks a registry for live instances and the load balancer picks a healthy one, so a single dead instance is invisible.
- Retry with backoff – the caller retries a bounded number of times with growing delays and jitter, absorbing transient blips without stampeding.
- Circuit breaker – once failures cross a threshold the breaker opens and fails fast without attempting the call, protecting both caller and callee.
- Fallback – when all else fails the gateway returns a cached value, a safe default, or a clear error so the client is never left hanging.
- Feature flags – operators can disable or degrade a feature at runtime without a redeploy, routing around a known-bad service.
- Timeout discipline – every downstream call has an aggressive timeout because a hang is the worst failure mode.
Imagine Rajesh goes to the loans counter at National Bank of India but the loan officer has stepped out. A well-run bank does not make him stand there forever. First the receptionist checks if another loan officer at a nearby counter is free, that is trying another instance. If the first officer is only briefly away, the receptionist waits a short, growing moment and checks again, that is retry with backoff. If the whole loans department is shut for the day, the receptionist stops sending people there and immediately says so, that is the circuit breaker. And she can hand Rajesh yesterday's printed interest-rate sheet so he is not empty-handed, that is a cached fallback.
Understanding the Concepts
What happens when the target service is down?
When a request reaches the API gateway and the target microservice is unreachable, the gateway must fail gracefully rather than hang or crash. The standard approach is a layered resilience toolkit where each layer handles a specific failure class. The layers must be ordered correctly: service discovery and load balancing first, then a short bounded retry, then a circuit breaker wrapping the whole thing, then a fallback. This ordering ensures that transient blips are absorbed silently, sustained outages trigger fast failure, and the client always receives a bounded response.
package com.bank.gateway;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import io.github.resilience4j.timelimiter.annotation.TimeLimiter;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class AccountGatewayService {
private final AccountServiceClient client;
private final BalanceCacheService cache;
public AccountGatewayService(AccountServiceClient client,
BalanceCacheService cache) {
this.client = client;
this.cache = cache;
}
@TimeLimiter(name = "accountService")
@Retry(name = "accountRetry")
@CircuitBreaker(name = "accountCb", fallbackMethod = "balanceFallback")
public BigDecimal getBalance(String accountId) {
return client.fetchBalance(accountId);
}
public BigDecimal balanceFallback(String accountId, Throwable t) {
// Return cached balance when the service is down
return cache.getLastKnownBalance(accountId);
}
}
Expected output (service healthy):
Balance for SAV2001: 150000.00
Expected output (service down, fallback):
Balance for SAV2001: 149500.00 (cached)
How does service discovery help?
Service discovery with client-side load balancing is the first line of defense. A service usually runs as several instances registered in a registry like Eureka or Consul. When the gateway needs the account service, it asks the registry for live instances and the load balancer picks one. If that instance is down, the next call naturally lands on another healthy instance, so a single dead instance causes almost no visible failure.
package com.bank.config;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;
@Configuration
@EnableDiscoveryClient
public class DiscoveryConfig {
@Bean
@LoadBalanced
public RestTemplate restTemplate() {
// The LoadBalanced annotation makes RestTemplate resolve
// service names through the registry and balance across instances
return new RestTemplate();
}
}
# Eureka dashboard shows:
account-service: 3 instances (UP)
192.168.1.10:8080 UP
192.168.1.11:8080 UP
192.168.1.12:8080 DOWN <-- load balancer skips this one
How does retry with exponential backoff work?
Retry with exponential backoff handles transient failures like a brief network blip or a momentary garbage-collection pause. The gateway retries a small, bounded number of times, waiting longer between each attempt, for example 100 milliseconds, then 200, then 400, with a little randomness (jitter) so many clients do not retry in lockstep. The key rule is that retries must be bounded and should only apply to idempotent operations, because retrying a money transfer blindly could double-charge Rajesh.
# application.yml - Resilience4j retry configuration
resilience4j:
retry:
instances:
accountRetry:
max-attempts: 3
wait-duration: 100ms
exponential-backoff-multiplier: 2
enable-exponential-backoff: true
enable-randomized-wait: true
randomized-wait-factor: 0.5
retry-exceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
ignore-exceptions:
- com.bank.exception.InsufficientFundsException
Retry timeline:
Attempt 1: immediate -> IOException
Attempt 2: wait ~150ms -> IOException
Attempt 3: wait ~300ms -> Success (balance returned)
What is a circuit breaker and how does it protect the system?
A circuit breaker watches the failure rate of calls to a downstream service. Once failures cross a configured threshold, the breaker "opens" and immediately fails fast without even attempting the call. This protects both the caller (no wasted threads or resources) and the struggling service (no pile-on while it recovers). After a cool-down period the breaker goes "half-open" and lets a few trial calls through. If they succeed it "closes" and resumes normal traffic. If they fail it reopens.
# application.yml - Resilience4j circuit breaker configuration
resilience4j:
circuitbreaker:
instances:
accountCb:
sliding-window-type: COUNT_BASED
sliding-window-size: 10
failure-rate-threshold: 50
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 3
minimum-number-of-calls: 5
record-exceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
Circuit breaker state transitions:
CLOSED (normal) -> 5 of 10 calls fail -> OPEN (fail fast)
OPEN -> 30 seconds pass -> HALF_OPEN (trial)
HALF_OPEN -> 3 trial calls pass -> CLOSED (normal)
HALF_OPEN -> 1 trial call fails -> OPEN (reopen)
What makes a good fallback?
The fallback decides what the client actually receives when all else fails. A good fallback turns a scary failure into a graceful degradation. The options are a cached value (last known balance), a sensible default (empty list), or a clear error with the right HTTP status code. The choice depends on the business context.
package com.bank.gateway;
import java.math.BigDecimal;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class BalanceCacheService {
private final Map<String, BigDecimal> cache = new ConcurrentHashMap<>();
public void updateCache(String accountId, BigDecimal balance) {
cache.put(accountId, balance);
}
public BigDecimal getLastKnownBalance(String accountId) {
BigDecimal cached = cache.get(accountId);
if (cached != null) {
return cached; // Stale but useful
}
// No cache entry -- return a clear indication
throw new ServiceUnavailableException(
"Account service unavailable and no cached balance for " + accountId
);
}
}
Fallback decision matrix:
Cached value exists -> return cached balance (stale but useful)
No cached value -> return 503 with Retry-After header
Write operation -> never use a fallback, return honest error
How do feature flags help during an outage?
Feature flags let operators change behavior at runtime without a deploy. If a downstream service is being drained for maintenance, an operator flips a flag so the gateway routes around it or returns a maintenance message, rather than relying on failures to trigger fallbacks. This gives proactive control over degradation.
package com.bank.gateway;
import java.math.BigDecimal;
public class AccountGatewayWithFeatureFlag {
private final FeatureFlagService flags;
private final AccountServiceClient client;
private final BalanceCacheService cache;
public AccountGatewayWithFeatureFlag(FeatureFlagService flags,
AccountServiceClient client,
BalanceCacheService cache) {
this.flags = flags;
this.client = client;
this.cache = cache;
}
public BigDecimal getBalance(String accountId) {
if (flags.isEnabled("account-service-maintenance")) {
// Proactively serve cached data during planned maintenance
return cache.getLastKnownBalance(accountId);
}
return client.fetchBalance(accountId);
}
}
# Operator toggles the flag during maintenance window:
curl -X POST http://config-server/flags/account-service-maintenance \
-d '{"enabled": true}'
# All gateway instances immediately serve cached balances
What is the correct ordering of resilience layers?
The layers must be ordered from outermost to innermost: timeout wraps everything, then circuit breaker, then retry, then the actual call. Service discovery and load balancing operate at the transport level. The fallback is attached to the circuit breaker. Getting the order wrong can cause retries inside an open circuit breaker (wasted effort) or timeouts that do not account for retry delays.
| Layer | Purpose | Position |
|---|---|---|
| Timeout | Bound total wait time | Outermost |
| Circuit breaker | Fail fast on sustained outage | Wraps retries |
| Retry | Absorb transient blips | Wraps call |
| Load balancer | Pick healthy instance | Transport level |
| Fallback | Return cached or default | Attached to breaker |
package com.bank.gateway;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import io.github.resilience4j.timelimiter.annotation.TimeLimiter;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.concurrent.CompletableFuture;
@Service
public class ResilientAccountService {
private final AccountServiceClient client;
public ResilientAccountService(AccountServiceClient client) {
this.client = client;
}
// Order: TimeLimiter (outermost) -> CircuitBreaker -> Retry (innermost)
@TimeLimiter(name = "accountTl")
@CircuitBreaker(name = "accountCb", fallbackMethod = "fallback")
@Retry(name = "accountRetry")
public CompletableFuture<BigDecimal> getBalance(String accountId) {
return CompletableFuture.supplyAsync(
() -> client.fetchBalance(accountId)
);
}
public CompletableFuture<BigDecimal> fallback(String accountId, Throwable t) {
return CompletableFuture.completedFuture(BigDecimal.ZERO);
}
}
Why is timeout discipline the most important piece?
Every downstream call needs an aggressive timeout because the worst failure is not an error but a hang. When a service hangs instead of failing, threads pile up waiting on a dead service and the gateway itself collapses. A proper timeout converts a hang into a fast, clean failure that the circuit breaker and fallback can handle.
package com.bank.config;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;
import java.time.Duration;
@Configuration
public class TimeoutConfig {
@Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder
.setConnectTimeout(Duration.ofMillis(500)) // Fast connect timeout
.setReadTimeout(Duration.ofSeconds(2)) // Aggressive read timeout
.build();
}
}
Without timeout:
Thread 1: waiting on dead service... 30s... 60s... pool exhausted
Thread 2: waiting on dead service... gateway unresponsive
With 2-second timeout:
Thread 1: timeout after 2s -> circuit breaker -> fallback -> thread freed
Thread 2: circuit breaker already open -> instant fallback -> thread freed
What are common mistakes when handling a down service?
The most common mistake is retrying non-idempotent operations. Retrying a transfer that already succeeded but whose response was lost can double-charge the customer. Another mistake is missing the timeout, which causes thread-pool exhaustion. A third is placing the circuit breaker inside the retry, which means the breaker opens and closes on every retry attempt rather than on the overall call.
package com.bank.gateway;
import java.math.BigDecimal;
public class WrongRetryExample {
// WRONG: retrying a non-idempotent transfer
public void transferWithRetry(String from, String to, BigDecimal amount) {
for (int attempt = 0; attempt < 3; attempt++) {
try {
// This could succeed on attempt 1, response lost,
// then attempt 2 charges Rajesh again!
transferService.transfer(from, to, amount);
return;
} catch (Exception e) {
// Blindly retry -- dangerous for writes!
}
}
}
}
package com.bank.gateway;
import java.math.BigDecimal;
public class CorrectRetryExample {
// CORRECT: use an idempotency key so retries are safe
public void transferWithIdempotency(String idempotencyKey,
String from, String to,
BigDecimal amount) {
// The server uses the idempotency key to detect duplicates
// and return the original result instead of charging again
transferService.transfer(idempotencyKey, from, to, amount);
}
}
Wrong: retry transfer without idempotency key -> double charge risk
Correct: retry with idempotency key -> server detects duplicate, returns original result
Visual Overview
Client Request
│
▼
┌─────────────┐
│ API Gateway │
└──────┬──────┘
│
▼
┌──────────────────┐
│ Service Discovery│
│ (Eureka/Consul) │
│ Pick healthy │
│ instance │
└──────┬───────────┘
│
▼
┌──────────────────┐ ┌───────────────┐
│ TimeLimiter │ │ │
│ (2s timeout) │ │ Fallback │
│ ┌──────────────┐ │ │ - cached val │
│ │Circuit Breaker│─┼──X──│ - default │
│ │ (fail fast) │ │ │ - 503 error │
│ │ ┌──────────┐ │ │ └───────────────┘
│ │ │ Retry │ │ │
│ │ │ (3x exp) │ │ │
│ │ │ ┌──────┐ │ │ │
│ │ │ │ Call │ │ │ │
│ │ │ └──────┘ │ │ │
│ │ └──────────┘ │ │
│ └──────────────┘ │
└──────────────────┘
│
▼
Target Service Instance
Key Takeaway
When a request reaches the gateway and the target service is down, the goal is to fail gracefully and in a bounded way, never to hang or cascade. The toolkit is layered and the order matters. Service discovery with client-side load balancing is the first layer, because most services run as several instances and the load balancer simply picks a healthy one. The second layer is a bounded retry with exponential backoff and jitter, which absorbs transient blips. Retries must be idempotent, because blindly retrying a money transfer could double-charge Rajesh. The third layer is the circuit breaker, which handles sustained outages by failing fast without even attempting the call, protecting both the gateway and the struggling service.
The fourth layer is the fallback, which decides what the client actually receives. Serving the last known balance from the cache turns a scary outage into a graceful degradation. Feature flags add proactive control, letting operators route around a service during planned maintenance without waiting for failures to trigger fallbacks. Timeout discipline ties everything together, because the worst failure mode is not an error but a hang where threads pile up waiting on a dead service and the gateway itself collapses.
In a real banking scenario at National Bank of India, the Balance Service had a bad deployment and started failing every request. Without protection, the gateway kept calling it, each call hung until timeout, threads piled up, and the whole gateway became unresponsive, breaking healthy features like viewing card details. With the layered resilience toolkit, the circuit breaker opened after five failures, the fallback served cached balances, and the gateway stayed responsive for all other features while the team fixed the deployment.
Remember one sentence: when the target is down, the gateway layers discovery, retry, circuit breaker, and fallback so it always returns a bounded, graceful response instead of hanging or cascading.
---