Definition
- Retry is the pattern of repeating a failed network request because the failure might be transient (timeout, 503, connection reset) and the next attempt might succeed.
- Exponential backoff is a retry delay strategy where the wait time doubles with each attempt (1s, 2s, 4s, 8s), giving the downstream service progressively more time to recover.
- Jitter is a random offset added to the backoff delay so that multiple clients retrying simultaneously do not create thundering herd load spikes.
- Idempotency key is a unique identifier (typically a UUID) generated by the client and sent with each request, allowing the server to detect and short-circuit duplicate requests.
- @Retryable is a Spring Retry annotation that automatically retries a method when it throws a specified exception, with configurable max attempts and backoff parameters.
Think of Rajesh at the National Bank of India ATM. He inserts his card and requests a 50,000 rupee withdrawal. The ATM sends the request to the bank server, but the network is slow and times out. The ATM screen shows an error. Rajesh tries again with the same card. Without idempotency, the bank deducts 50,000 twice because each request looks new. With idempotency, the ATM generates a unique transaction ID for each withdrawal attempt. The bank checks: has this transaction ID been processed? Yes, so it returns the previous result without deducting again.
Understanding the Concepts
What is a retry in microservices?
A retry is when a client repeats a failed request to a service, hoping the failure was transient (temporary). Transient failures include network timeouts, TCP connection resets, HTTP 503 Service Unavailable, and brief database connection pool exhaustion. These failures resolve on their own within seconds.
The danger of retrying is that if the original request succeeded on the server but the response was lost, the retry will execute the operation a second time. For a payment service, this means charging the customer twice. This is why retries must always be paired with idempotency.
package com.bank.retry;
public class RetryConceptDemo {
sealed interface RetryDecision permits ShouldRetry, ShouldNotRetry {}
record ShouldRetry(int httpStatus, String reason) implements RetryDecision {}
record ShouldNotRetry(int httpStatus, String reason) implements RetryDecision {}
static RetryDecision decideRetry(int httpStatus) {
return switch (httpStatus) {
case 408 -> new ShouldRetry(408, "Request Timeout: server was slow, retry");
case 429 -> new ShouldRetry(429, "Too Many Requests: back off and retry");
case 500 -> new ShouldRetry(500, "Internal Server Error: might be transient");
case 502 -> new ShouldRetry(502, "Bad Gateway: upstream issue, retry");
case 503 -> new ShouldRetry(503, "Service Unavailable: server overloaded, retry");
case 400 -> new ShouldNotRetry(400, "Bad Request: fix the payload, do not retry");
case 401 -> new ShouldNotRetry(401, "Unauthorized: fix credentials, do not retry");
case 404 -> new ShouldNotRetry(404, "Not Found: resource does not exist, do not retry");
case 409 -> new ShouldNotRetry(409, "Conflict: duplicate, do not retry");
case 422 -> new ShouldNotRetry(422, "Unprocessable: business rule violation, do not retry");
default -> new ShouldNotRetry(httpStatus, "Unknown status, do not retry");
};
}
public static void main(String[] args) {
var statuses = java.util.List.of(408, 429, 500, 503, 400, 404, 409, 422);
for (var status : statuses) {
var decision = decideRetry(status);
String label = switch (decision) {
case ShouldRetry r -> "RETRY " + r.httpStatus() + " " + r.reason();
case ShouldNotRetry n -> "NO RETRY " + n.httpStatus() + " " + n.reason();
};
System.out.println(label);
}
}
}
Expected output:
RETRY 408 Request Timeout: server was slow, retry
RETRY 429 Too Many Requests: back off and retry
RETRY 500 Internal Server Error: might be transient
RETRY 503 Service Unavailable: server overloaded, retry
NO RETRY 400 Bad Request: fix the payload, do not retry
NO RETRY 404 Not Found: resource does not exist, do not retry
NO RETRY 409 Conflict: duplicate, do not retry
NO RETRY 422 Unprocessable: business rule violation, do not retry
What is exponential backoff with jitter?
Exponential backoff means increasing the wait time between retries exponentially: 1 second, 2 seconds, 4 seconds, 8 seconds, and so on. This gives the failing service time to recover.
Jitter adds a random component to the wait time. Without jitter, if 100 clients all started retrying at the same time (after a brief outage), they would all retry at the exact same intervals, creating thundering herd spikes that overload the recovering service again. With jitter, each client waits a slightly different amount of time, spreading the load.
The formula is: delay = min(baseDelay * 2^attempt + random(0, jitterRange), maxDelay)
package com.bank.retry;
import java.util.Random;
public class ExponentialBackoffDemo {
record RetryDelay(int attempt, long baseDelayMs, long jitterMs, long totalDelayMs) {}
public static void main(String[] args) {
long baseDelay = 1000; // 1 second
long maxDelay = 30000; // 30 seconds cap
var random = new Random(42); // Fixed seed for reproducible output
System.out.println("Exponential backoff with jitter:");
System.out.println("Base delay: " + baseDelay + "ms, Max delay: " + maxDelay + "ms");
System.out.println();
for (int attempt = 1; attempt <= 5; attempt++) {
long exponentialDelay = baseDelay * (long) Math.pow(2, attempt - 1);
long jitter = random.nextLong(0, baseDelay); // Jitter between 0 and baseDelay
long totalDelay = Math.min(exponentialDelay + jitter, maxDelay);
var delay = new RetryDelay(attempt, exponentialDelay, jitter, totalDelay);
System.out.println(" Attempt " + delay.attempt()
+ ": base=" + delay.baseDelayMs() + "ms"
+ ", jitter=" + delay.jitterMs() + "ms"
+ ", total=" + delay.totalDelayMs() + "ms");
}
System.out.println();
System.out.println("Without jitter: all 100 clients retry at 1s, 2s, 4s, 8s, 16s");
System.out.println(" -> thundering herd at each retry interval");
System.out.println("With jitter: client A retries at 1.3s, B at 1.7s, C at 0.9s...");
System.out.println(" -> load is spread out, server recovers");
}
}
Expected output:
Exponential backoff with jitter:
Base delay: 1000ms, Max delay: 30000ms
Attempt 1: base=1000ms, jitter=530ms, total=1530ms
Attempt 2: base=2000ms, jitter=444ms, total=2444ms
Attempt 3: base=4000ms, jitter=287ms, total=4287ms
Attempt 4: base=8000ms, jitter=811ms, total=8811ms
Attempt 5: base=16000ms, jitter=155ms, total=16155ms
Without jitter: all 100 clients retry at 1s, 2s, 4s, 8s, 16s
-> thundering herd at each retry interval
With jitter: client A retries at 1.3s, B at 1.7s, C at 0.9s...
-> load is spread out, server recovers
What is idempotency and why is it required for retries?
An operation is idempotent if executing it multiple times produces the same result as executing it once. For a payment debit of 50,000 rupees, executing the debit once should subtract 50,000. Executing the same debit request twice (because the client retried after a timeout) should still subtract only 50,000 total, not 100,000.
The standard approach is an idempotency key: a unique identifier generated by the client and sent with the request. The server stores this key in a database with a unique constraint. Before processing a request, the server checks if the key already exists. If it does, the request is a duplicate and the server returns the stored result without processing again.
| Scenario | Without Idempotency | With Idempotency |
|---|---|---|
| First request succeeds | Debit 50,000. Balance: 450,000 | Debit 50,000. Balance: 450,000. Store key. |
| Response lost, client retries | Debit 50,000 again. Balance: 400,000 (WRONG) | Key exists, return stored result. Balance: 450,000 (CORRECT) |
| Client retries a third time | Debit 50,000 again. Balance: 350,000 (WRONG) | Key exists, return stored result. Balance: 450,000 (CORRECT) |
package com.bank.retry;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.Map;
public class IdempotencyDemo {
record PaymentResult(String idempotencyKey, BigDecimal amountDebited, BigDecimal newBalance, boolean wasDuplicate) {}
static Map<String, PaymentResult> processedKeys = new HashMap<>();
static BigDecimal balance = new BigDecimal("500000.00");
static PaymentResult processPayment(String idempotencyKey, BigDecimal amount) {
// Check if this request was already processed
if (processedKeys.containsKey(idempotencyKey)) {
var stored = processedKeys.get(idempotencyKey);
System.out.println(" DUPLICATE detected for key=" + idempotencyKey + ", returning stored result");
return new PaymentResult(idempotencyKey, stored.amountDebited(), stored.newBalance(), true);
}
// Process the payment
balance = balance.subtract(amount);
var result = new PaymentResult(idempotencyKey, amount, balance, false);
processedKeys.put(idempotencyKey, result);
System.out.println(" PROCESSED: key=" + idempotencyKey + ", debited=" + amount + ", newBalance=" + balance);
return result;
}
public static void main(String[] args) {
var key = "PAY-2024-001";
var amount = new BigDecimal("50000.00");
System.out.println("Balance before: " + balance);
System.out.println();
System.out.println("Attempt 1 (original request):");
var r1 = processPayment(key, amount);
System.out.println(" Result: debited=" + r1.amountDebited() + ", balance=" + r1.newBalance() + ", duplicate=" + r1.wasDuplicate());
System.out.println();
System.out.println("Attempt 2 (retry after timeout):");
var r2 = processPayment(key, amount);
System.out.println(" Result: debited=" + r2.amountDebited() + ", balance=" + r2.newBalance() + ", duplicate=" + r2.wasDuplicate());
System.out.println();
System.out.println("Attempt 3 (another retry):");
var r3 = processPayment(key, amount);
System.out.println(" Result: debited=" + r3.amountDebited() + ", balance=" + r3.newBalance() + ", duplicate=" + r3.wasDuplicate());
System.out.println();
System.out.println("Final balance: " + balance + " (debited only once, not three times)");
}
}
Expected output:
Balance before: 500000.00
Attempt 1 (original request):
PROCESSED: key=PAY-2024-001, debited=50000.00, newBalance=450000.00
Result: debited=50000.00, balance=450000.00, duplicate=false
Attempt 2 (retry after timeout):
DUPLICATE detected for key=PAY-2024-001, returning stored result
Result: debited=50000.00, balance=450000.00, duplicate=true
Attempt 3 (another retry):
DUPLICATE detected for key=PAY-2024-001, returning stored result
Result: debited=50000.00, balance=450000.00, duplicate=true
Final balance: 450000.00 (debited only once, not three times)
How is retry different from circuit breaker?
Retry and circuit breaker are complementary resilience patterns. Retry repeats a failed request hoping it succeeds on the next attempt. Circuit breaker stops sending requests to a service that is repeatedly failing, giving it time to recover.
| Feature | Retry | Circuit Breaker |
|---|---|---|
| Purpose | Handle transient failures | Prevent cascading failures |
| When it activates | On each individual failure | After N consecutive failures |
| Behavior | Sends the request again | Rejects requests immediately (fast-fail) |
| Timeout | Adds exponential backoff delay | Opens for a fixed duration, then half-opens |
| Risk | Can overload a struggling service | Can reject requests even after service recovers |
| Works best with | Idempotent operations | Fallback responses or error pages |
| Spring library | spring-retry @Retryable | Resilience4j @CircuitBreaker |
package com.bank.retry;
public class RetryVsCircuitBreaker {
enum CircuitState { CLOSED, OPEN, HALF_OPEN }
public static void main(String[] args) {
System.out.println("=== Retry Pattern ===");
System.out.println("Request 1: FAIL (503) -> wait 1s -> retry");
System.out.println("Request 1: FAIL (503) -> wait 2s -> retry");
System.out.println("Request 1: SUCCESS (200) -> done");
System.out.println("Total time: ~3 seconds for 1 request");
System.out.println();
System.out.println("=== Circuit Breaker Pattern ===");
System.out.println("State: CLOSED (normal operation)");
System.out.println("Request 1: FAIL -> failure count=1");
System.out.println("Request 2: FAIL -> failure count=2");
System.out.println("Request 3: FAIL -> failure count=3 (threshold reached)");
System.out.println("State: OPEN (reject all requests for 30 seconds)");
System.out.println("Request 4: REJECTED immediately (no network call)");
System.out.println("Request 5: REJECTED immediately");
System.out.println("...30 seconds later...");
System.out.println("State: HALF_OPEN (allow one test request)");
System.out.println("Request 6: SUCCESS -> State: CLOSED (back to normal)");
System.out.println();
System.out.println("=== Best Practice: Use Both Together ===");
System.out.println("Circuit Breaker wraps Retry:");
System.out.println(" if (circuit is CLOSED) { try with retries } else { fast-fail }");
}
}
Expected output:
=== Retry Pattern ===
Request 1: FAIL (503) -> wait 1s -> retry
Request 1: FAIL (503) -> wait 2s -> retry
Request 1: SUCCESS (200) -> done
Total time: ~3 seconds for 1 request
=== Circuit Breaker Pattern ===
State: CLOSED (normal operation)
Request 1: FAIL -> failure count=1
Request 2: FAIL -> failure count=2
Request 3: FAIL -> failure count=3 (threshold reached)
State: OPEN (reject all requests for 30 seconds)
Request 4: REJECTED immediately (no network call)
Request 5: REJECTED immediately
...30 seconds later...
State: HALF_OPEN (allow one test request)
Request 6: SUCCESS -> State: CLOSED (back to normal)
=== Best Practice: Use Both Together ===
Circuit Breaker wraps Retry:
if (circuit is CLOSED) { try with retries } else { fast-fail }
What is a complete banking project with retries and idempotency?
This project has a payment-service that processes payments. For each payment, it calls an external fraud-check-service via REST. The fraud-check-service is unreliable (simulated with WireMock in tests). The payment-service uses Spring Retry with exponential backoff and jitter, plus an idempotency key stored in a database to prevent duplicate payments.
Project structure
payment-retry-service/
├── pom.xml
├── docker-compose.yml
├── init-db.sql
└── src/
├── main/
│ ├── java/com/bank/payment/
│ │ ├── PaymentServiceApp.java
│ │ ├── controller/
│ │ │ └── PaymentController.java
│ │ ├── service/
│ │ │ └── PaymentService.java
│ │ ├── client/
│ │ │ └── FraudCheckClient.java
│ │ ├── repository/
│ │ │ ├── PaymentRepository.java
│ │ │ └── IdempotencyRepository.java
│ │ ├── model/
│ │ │ ├── Payment.java
│ │ │ └── IdempotencyRecord.java
│ │ ├── dto/
│ │ │ ├── PaymentRequest.java
│ │ │ └── FraudCheckResponse.java
│ │ ├── config/
│ │ │ ├── RetryConfig.java
│ │ │ └── RestClientConfig.java
│ │ └── exception/
│ │ ├── DuplicatePaymentException.java
│ │ ├── FraudCheckFailedException.java
│ │ └── GlobalExceptionHandler.java
│ └── resources/
│ └── application.yml
└── test/
└── java/com/bank/payment/
└── PaymentRetryIntegrationTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><parentsyn-kw">>
syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>spring-boot-starter-parentsyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>3.4.1syn-kw"></versionsyn-kw">>
syn-kw"></parentsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>payment-retry-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>spring-boot-starter-websyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>spring-boot-starter-data-jpasyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.springframework.retrysyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>spring-retrysyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.springframeworksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>spring-aspectssyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.postgresqlsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>postgresqlsyn-kw"></artifactIdsyn-kw">>
syn-kw"><scopesyn-kw">>runtimesyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>spring-boot-starter-testsyn-kw"></artifactIdsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.wiremocksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>wiremock-standalonesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>3.5.4syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
init-db.sql
CREATE TABLE IF NOT EXISTS payments (
id BIGSERIAL PRIMARY KEY,
idempotency_key VARCHAR(50) NOT NULL,
from_account VARCHAR(20) NOT NULL,
to_account VARCHAR(20) NOT NULL,
amount NUMERIC(15,2) NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
fraud_check_result VARCHAR(20),
retry_count INT NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS idempotency_records (
id BIGSERIAL PRIMARY KEY,
idempotency_key VARCHAR(50) NOT NULL UNIQUE,
response_status VARCHAR(20) NOT NULL,
response_body TEXT,
created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
Payment.java
package com.bank.payment.model;
import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;
import java.math.BigDecimal;
import java.time.LocalDateTime;
@Entity
@Table(name = "payments")
public class Payment {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
@Column(name = "idempotency_key", nullable = false)
private String idempotencyKey;
@Column(name = "from_account", nullable = false)
private String fromAccount;
@Column(name = "to_account", nullable = false)
private String toAccount;
@Column(nullable = false)
private BigDecimal amount;
@Column(nullable = false)
private String status;
@Column(name = "fraud_check_result")
private String fraudCheckResult;
@Column(name = "retry_count")
private int retryCount;
@Column(name = "created_at", nullable = false)
private LocalDateTime createdAt = LocalDateTime.now();
public Long getId() { return id; }
public String getIdempotencyKey() { return idempotencyKey; }
public void setIdempotencyKey(String key) { this.idempotencyKey = key; }
public String getFromAccount() { return fromAccount; }
public void setFromAccount(String fromAccount) { this.fromAccount = fromAccount; }
public String getToAccount() { return toAccount; }
public void setToAccount(String toAccount) { this.toAccount = toAccount; }
public BigDecimal getAmount() { return amount; }
public void setAmount(BigDecimal amount) { this.amount = amount; }
public String getStatus() { return status; }
public void setStatus(String status) { this.status = status; }
public String getFraudCheckResult() { return fraudCheckResult; }
public void setFraudCheckResult(String result) { this.fraudCheckResult = result; }
public int getRetryCount() { return retryCount; }
public void setRetryCount(int count) { this.retryCount = count; }
}
IdempotencyRecord.java
package com.bank.payment.model;
import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;
import java.time.LocalDateTime;
@Entity
@Table(name = "idempotency_records")
public class IdempotencyRecord {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
@Column(name = "idempotency_key", nullable = false, unique = true)
private String idempotencyKey;
@Column(name = "response_status", nullable = false)
private String responseStatus;
@Column(name = "response_body")
private String responseBody;
@Column(name = "created_at", nullable = false)
private LocalDateTime createdAt = LocalDateTime.now();
public Long getId() { return id; }
public String getIdempotencyKey() { return idempotencyKey; }
public void setIdempotencyKey(String key) { this.idempotencyKey = key; }
public String getResponseStatus() { return responseStatus; }
public void setResponseStatus(String status) { this.responseStatus = status; }
public String getResponseBody() { return responseBody; }
public void setResponseBody(String body) { this.responseBody = body; }
}
PaymentRequest.java and FraudCheckResponse.java
package com.bank.payment.dto;
import java.math.BigDecimal;
public record PaymentRequest(
String idempotencyKey,
String fromAccount,
String toAccount,
BigDecimal amount,
String description
) {}
package com.bank.payment.dto;
public record FraudCheckResponse(String status, String riskLevel, String message) {}
PaymentRepository.java and IdempotencyRepository.java
package com.bank.payment.repository;
import com.bank.payment.model.Payment;
import org.springframework.data.jpa.repository.JpaRepository;
public interface PaymentRepository extends JpaRepository<Payment, Long> {}
package com.bank.payment.repository;
import com.bank.payment.model.IdempotencyRecord;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.Optional;
public interface IdempotencyRepository extends JpaRepository<IdempotencyRecord, Long> {
Optional<IdempotencyRecord> findByIdempotencyKey(String key);
}
RetryConfig.java
package com.bank.payment.config;
import org.springframework.context.annotation.Configuration;
import org.springframework.retry.annotation.EnableRetry;
@Configuration
@EnableRetry
public class RetryConfig {
// @EnableRetry activates @Retryable annotations across the application.
// Retry parameters are configured on each @Retryable method.
}
RestClientConfig.java
package com.bank.payment.config;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestClient;
@Configuration
public class RestClientConfig {
@Value("\${fraud.check.service.url:http://localhost:8090}")
private String fraudCheckUrl;
@Bean
public RestClient fraudCheckRestClient() {
var factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(2000);
factory.setReadTimeout(3000);
return RestClient.builder()
.baseUrl(fraudCheckUrl)
.requestFactory(factory)
.build();
}
}
FraudCheckClient.java (with @Retryable)
package com.bank.payment.client;
import com.bank.payment.dto.FraudCheckResponse;
import com.bank.payment.exception.FraudCheckFailedException;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Recover;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import org.springframework.web.client.RestClientException;
import java.math.BigDecimal;
import java.util.Map;
import java.util.logging.Logger;
@Component
public class FraudCheckClient {
private static final Logger LOG = Logger.getLogger(FraudCheckClient.class.getName());
private final RestClient restClient;
public FraudCheckClient(RestClient fraudCheckRestClient) {
this.restClient = fraudCheckRestClient;
}
@Retryable(
retryFor = RestClientException.class,
maxAttempts = 3,
backoff = @Backoff(delay = 1000, multiplier = 2.0, random = true)
)
public FraudCheckResponse checkFraud(String accountId, BigDecimal amount) {
LOG.info("Calling fraud-check-service for account=" + accountId + ", amount=" + amount);
try {
var body = Map.of("accountId", accountId, "amount", amount);
return restClient.post()
.uri("/fraud/check")
.body(body)
.retrieve()
.body(FraudCheckResponse.class);
} catch (RestClientException ex) {
LOG.warning("Fraud check failed: " + ex.getMessage() + ". Will retry if attempts remain.");
throw ex;
}
}
@Recover
public FraudCheckResponse recoverFraudCheck(RestClientException ex, String accountId, BigDecimal amount) {
LOG.severe("All retry attempts exhausted for fraud check. account=" + accountId);
throw new FraudCheckFailedException("Fraud check failed after all retries for " + accountId, ex);
}
}
PaymentService.java (with idempotency check)
package com.bank.payment.service;
import com.bank.payment.client.FraudCheckClient;
import com.bank.payment.dto.FraudCheckResponse;
import com.bank.payment.exception.DuplicatePaymentException;
import com.bank.payment.model.IdempotencyRecord;
import com.bank.payment.model.Payment;
import com.bank.payment.repository.IdempotencyRepository;
import com.bank.payment.repository.PaymentRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
import java.util.logging.Logger;
@Service
public class PaymentService {
private static final Logger LOG = Logger.getLogger(PaymentService.class.getName());
private final PaymentRepository paymentRepository;
private final IdempotencyRepository idempotencyRepository;
private final FraudCheckClient fraudCheckClient;
public PaymentService(PaymentRepository paymentRepository,
IdempotencyRepository idempotencyRepository,
FraudCheckClient fraudCheckClient) {
this.paymentRepository = paymentRepository;
this.idempotencyRepository = idempotencyRepository;
this.fraudCheckClient = fraudCheckClient;
}
@Transactional
public Payment processPayment(String idempotencyKey, String from, String to, BigDecimal amount) {
// Step 1: Idempotency check
var existing = idempotencyRepository.findByIdempotencyKey(idempotencyKey);
if (existing.isPresent()) {
LOG.info("Duplicate payment detected: key=" + idempotencyKey);
throw new DuplicatePaymentException(idempotencyKey);
}
// Step 2: Create payment record
var payment = new Payment();
payment.setIdempotencyKey(idempotencyKey);
payment.setFromAccount(from);
payment.setToAccount(to);
payment.setAmount(amount);
payment.setStatus("PENDING");
paymentRepository.save(payment);
// Step 3: Fraud check (with automatic retries via @Retryable)
FraudCheckResponse fraudResult = fraudCheckClient.checkFraud(from, amount);
payment.setFraudCheckResult(fraudResult.status());
if ("REJECTED".equals(fraudResult.status())) {
payment.setStatus("FRAUD_REJECTED");
paymentRepository.save(payment);
} else {
payment.setStatus("APPROVED");
paymentRepository.save(payment);
}
// Step 4: Record idempotency key
var record = new IdempotencyRecord();
record.setIdempotencyKey(idempotencyKey);
record.setResponseStatus(payment.getStatus());
record.setResponseBody(payment.getId().toString());
idempotencyRepository.save(record);
LOG.info("Payment processed: key=" + idempotencyKey + ", status=" + payment.getStatus());
return payment;
}
}
PaymentController.java
package com.bank.payment.controller;
import com.bank.payment.dto.PaymentRequest;
import com.bank.payment.model.Payment;
import com.bank.payment.service.PaymentService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/payments")
public class PaymentController {
private final PaymentService paymentService;
public PaymentController(PaymentService paymentService) {
this.paymentService = paymentService;
}
@PostMapping
public ResponseEntity<Payment> createPayment(@RequestBody PaymentRequest request) {
var payment = paymentService.processPayment(
request.idempotencyKey(), request.fromAccount(),
request.toAccount(), request.amount());
return ResponseEntity.ok(payment);
}
}
Exception classes
package com.bank.payment.exception;
public class DuplicatePaymentException extends RuntimeException {
public DuplicatePaymentException(String idempotencyKey) {
super("Duplicate payment: idempotency key " + idempotencyKey + " already processed");
}
}
package com.bank.payment.exception;
public class FraudCheckFailedException extends RuntimeException {
public FraudCheckFailedException(String message, Throwable cause) {
super(message, cause);
}
}
GlobalExceptionHandler.java
package com.bank.payment.exception;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import java.util.Map;
@RestControllerAdvice
public class GlobalExceptionHandler {
@ExceptionHandler(DuplicatePaymentException.class)
public ResponseEntity<Map<String, String>> handleDuplicate(DuplicatePaymentException ex) {
return ResponseEntity.status(HttpStatus.CONFLICT)
.body(Map.of("error", ex.getMessage()));
}
@ExceptionHandler(FraudCheckFailedException.class)
public ResponseEntity<Map<String, String>> handleFraudCheckFailed(FraudCheckFailedException ex) {
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.body(Map.of("error", ex.getMessage()));
}
}
application.yml
server:
port: 8083
spring:
application:
name: payment-retry-service
datasource:
url: jdbc:postgresql://localhost:5432/bank
username: bank
password: bank123
jpa:
hibernate:
ddl-auto: none
fraud:
check:
service:
url: http://localhost:8090
How to run
# Step 1: Start PostgreSQL and initialize tables
docker-compose up -d postgres
# Step 2: Build the project
mvn clean package -DskipTests
# Step 3: Start the application
java -jar target/payment-retry-service-1.0.0.jar
# Step 4: Send a payment request
curl -s -X POST http://localhost:8083/payments \
-H "Content-Type: application/json" \
-d '{"idempotencyKey":"PAY-001","fromAccount":"SAV2001","toAccount":"SAV2002","amount":75000,"description":"rent"}'
# Step 5: Send the SAME request again (duplicate)
curl -s -X POST http://localhost:8083/payments \
-H "Content-Type: application/json" \
-d '{"idempotencyKey":"PAY-001","fromAccount":"SAV2001","toAccount":"SAV2002","amount":75000,"description":"rent"}'
Expected output (first request):
{"id":1,"idempotencyKey":"PAY-001","fromAccount":"SAV2001","toAccount":"SAV2002",
"amount":75000.00,"status":"APPROVED","fraudCheckResult":"CLEAN","retryCount":0}
Expected output (duplicate request):
HTTP 409 Conflict
{"error":"Duplicate payment: idempotency key PAY-001 already processed"}
What are common mistakes with retries?
Wrong: retry without idempotency causes double charge
package com.bank.payment.wrong;
import java.math.BigDecimal;
public class WrongRetryWithoutIdempotency {
static BigDecimal balance = new BigDecimal("500000.00");
static void processPayment(BigDecimal amount) {
balance = balance.subtract(amount);
System.out.println(" Debited " + amount + ", new balance: " + balance);
}
public static void main(String[] args) {
var amount = new BigDecimal("50000.00");
System.out.println("WRONG: Retry without idempotency");
System.out.println("Balance before: " + balance);
System.out.println("Attempt 1: server processes, but response is lost");
processPayment(amount);
System.out.println("Attempt 2: client retries, server processes AGAIN");
processPayment(amount);
System.out.println("Attempt 3: client retries again, server processes AGAIN");
processPayment(amount);
System.out.println("Final balance: " + balance);
System.out.println("Expected: 450000.00, Actual: " + balance + " (CUSTOMER CHARGED 3x!)");
}
}
Expected output:
WRONG: Retry without idempotency
Balance before: 500000.00
Attempt 1: server processes, but response is lost
Debited 50000.00, new balance: 450000.00
Attempt 2: client retries, server processes AGAIN
Debited 50000.00, new balance: 400000.00
Attempt 3: client retries again, server processes AGAIN
Debited 50000.00, new balance: 350000.00
Final balance: 350000.00
Expected: 450000.00, Actual: 350000.00 (CUSTOMER CHARGED 3x!)
Correct: retry with idempotency key prevents duplicate charges
package com.bank.payment.correct;
import java.math.BigDecimal;
import java.util.HashSet;
import java.util.Set;
public class CorrectRetryWithIdempotency {
static BigDecimal balance = new BigDecimal("500000.00");
static Set<String> processedKeys = new HashSet<>();
static boolean processPayment(String idempotencyKey, BigDecimal amount) {
if (processedKeys.contains(idempotencyKey)) {
System.out.println(" DUPLICATE key=" + idempotencyKey + ", returning stored result");
return true; // Return success without processing again
}
balance = balance.subtract(amount);
processedKeys.add(idempotencyKey);
System.out.println(" Debited " + amount + ", new balance: " + balance);
return true;
}
public static void main(String[] args) {
var amount = new BigDecimal("50000.00");
System.out.println("CORRECT: Retry with idempotency key");
System.out.println("Balance before: " + balance);
System.out.println("Attempt 1: server processes, response is lost");
processPayment("PAY-001", amount);
System.out.println("Attempt 2: client retries with same key");
processPayment("PAY-001", amount);
System.out.println("Attempt 3: client retries again with same key");
processPayment("PAY-001", amount);
System.out.println("Final balance: " + balance + " (debited only ONCE)");
}
}
Expected output:
CORRECT: Retry with idempotency key
Balance before: 500000.00
Attempt 1: server processes, response is lost
Debited 50000.00, new balance: 450000.00
Attempt 2: client retries with same key
DUPLICATE key=PAY-001, returning stored result
Attempt 3: client retries again with same key
DUPLICATE key=PAY-001, returning stored result
Final balance: 450000.00 (debited only ONCE)
How do you test retries with WireMock?
WireMock can simulate transient failures by returning different responses for consecutive calls to the same endpoint. The first two calls return HTTP 503 (Service Unavailable), and the third call returns HTTP 200 with a success body. This tests that your retry logic works correctly.
package com.bank.payment;
import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.stubbing.Scenario;
import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
import static com.github.tomakehurst.wiremock.client.WireMock.post;
import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
public class RetryWireMockTest {
public static void main(String[] args) {
var wireMock = new WireMockServer(8090);
wireMock.start();
// First call: return 503
wireMock.stubFor(post(urlEqualTo("/fraud/check"))
.inScenario("retry-test")
.whenScenarioStateIs(Scenario.STARTED)
.willReturn(aResponse().withStatus(503).withBody("Service Unavailable"))
.willSetStateTo("ATTEMPT_2"));
// Second call: return 503
wireMock.stubFor(post(urlEqualTo("/fraud/check"))
.inScenario("retry-test")
.whenScenarioStateIs("ATTEMPT_2")
.willReturn(aResponse().withStatus(503).withBody("Service Unavailable"))
.willSetStateTo("ATTEMPT_3"));
// Third call: return 200
wireMock.stubFor(post(urlEqualTo("/fraud/check"))
.inScenario("retry-test")
.whenScenarioStateIs("ATTEMPT_3")
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "application/json")
.withBody("{\"status\":\"CLEAN\",\"riskLevel\":\"LOW\",\"message\":\"No fraud detected\"}")));
System.out.println("WireMock started on port 8090");
System.out.println("Scenario: 2 failures then 1 success");
System.out.println(" Call 1: HTTP 503 (triggers retry)");
System.out.println(" Call 2: HTTP 503 (triggers retry)");
System.out.println(" Call 3: HTTP 200 {status:CLEAN}");
System.out.println();
System.out.println("With @Retryable(maxAttempts=3, backoff=@Backoff(delay=1000, multiplier=2)):");
System.out.println(" Attempt 1: FAIL (503), wait ~1s");
System.out.println(" Attempt 2: FAIL (503), wait ~2s");
System.out.println(" Attempt 3: SUCCESS (200), fraud check CLEAN");
System.out.println(" Total time: ~3 seconds");
wireMock.stop();
}
}
Expected output:
WireMock started on port 8090
Scenario: 2 failures then 1 success
Call 1: HTTP 503 (triggers retry)
Call 2: HTTP 503 (triggers retry)
Call 3: HTTP 200 {status:CLEAN}
With @Retryable(maxAttempts=3, backoff=@Backoff(delay=1000, multiplier=2)):
Attempt 1: FAIL (503), wait ~1s
Attempt 2: FAIL (503), wait ~2s
Attempt 3: SUCCESS (200), fraud check CLEAN
Total time: ~3 seconds
What should a senior or principal engineer know?
At the senior level, you must understand the tradeoffs of retry strategies:
- Retry budget: limit the total number of retries per second across all clients. If you have 100 instances each retrying 3 times, you send 300 requests to a struggling service. A global retry budget caps the total.
- Retry amplification: in a chain of services (A calls B calls C), if each service retries 3 times, a single failure in C can generate 3 x 3 = 9 requests. Keep retry counts low (2-3 max) and only retry at the outermost caller.
- Idempotency key TTL: idempotency records should expire. A key from 30 days ago should not block a new request. Use a TTL of 24-48 hours in the database.
- Idempotency scope: the key should be scoped to the operation, not just the request. If the same amount from the same account with the same description is sent twice, it could be two legitimate payments. The client must generate a unique key for each intended payment.
package com.bank.payment;
public class SeniorRetryKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal retry and idempotency checklist:");
System.out.println(" 1. Always pair retries with idempotency keys");
System.out.println(" 2. Use exponential backoff + jitter, never fixed delay");
System.out.println(" 3. Only retry on transient errors (5xx, timeout), not 4xx");
System.out.println(" 4. Max 2-3 retry attempts per call");
System.out.println(" 5. Add retry budget to prevent retry storms");
System.out.println(" 6. Beware retry amplification in service chains");
System.out.println(" 7. Set TTL on idempotency keys (24-48 hours)");
System.out.println(" 8. Client generates the idempotency key, not the server");
System.out.println(" 9. Store idempotency key + result in the same transaction");
System.out.println(" 10. Log every retry attempt with correlation ID for debugging");
}
}
Expected output:
Senior/Principal retry and idempotency checklist:
1. Always pair retries with idempotency keys
2. Use exponential backoff + jitter, never fixed delay
3. Only retry on transient errors (5xx, timeout), not 4xx
4. Max 2-3 retry attempts per call
5. Add retry budget to prevent retry storms
6. Beware retry amplification in service chains
7. Set TTL on idempotency keys (24-48 hours)
8. Client generates the idempotency key, not the server
9. Store idempotency key + result in the same transaction
10. Log every retry attempt with correlation ID for debugging
What are common interview follow-up questions?
- What happens if the idempotency key table is in a different database than the payment table? (You lose atomicity. If the payment is saved but the idempotency record is not, a retry will process the payment again. Always store both in the same database and the same transaction.)
- Should you retry on HTTP 400 Bad Request? (No. A 400 means the request itself is invalid. Retrying the same invalid request will always fail. Fix the request, do not retry.)
- What is the difference between @Retryable and Resilience4j Retry? (@Retryable is from Spring Retry and uses AOP proxies. Resilience4j is a standalone library with decorators. Resilience4j offers more features like retry rate limiting and better metrics integration.)
- How do you handle a retry that exceeds the timeout budget? (Set a total timeout that includes all retry delays. If the total budget is 10 seconds and the first two attempts used 6 seconds, the third attempt only has 4 seconds left.)
- Can you use idempotency with async messaging? (Yes. The message ID or a custom idempotency key in the message header serves the same purpose. The consumer checks the key before processing.)
Visual Overview
Retry with Exponential Backoff and Idempotency Flow
=====================================================
Client payment-service fraud-check-service
│ │ │
│ POST /payments │ │
│ Idempotency-Key: PAY-001 │ │
│───────────────────────────>│ │
│ │ │
│ │ Check idempotency table │
│ │ Key PAY-001 not found │
│ │ │
│ │ POST /fraud/check │
│ │─────────────────────────────>│
│ │ 503 Unavailable │
│ │<─────────────────────────────│
│ │ │
│ │ Wait 1s (backoff + jitter) │
│ │ │
│ │ POST /fraud/check (retry 1) │
│ │─────────────────────────────>│
│ │ 503 Unavailable │
│ │<─────────────────────────────│
│ │ │
│ │ Wait 2s (backoff + jitter) │
│ │ │
│ │ POST /fraud/check (retry 2) │
│ │─────────────────────────────>│
│ │ 200 OK {CLEAN} │
│ │<─────────────────────────────│
│ │ │
│ │ Save payment + idem key │
│ │ (same DB transaction) │
│ │ │
│ 200 OK {APPROVED} │ │
│<───────────────────────────│ │
│ │ │
│ POST /payments (retry) │ │
│ Idempotency-Key: PAY-001 │ │
│───────────────────────────>│ │
│ │ Key PAY-001 FOUND │
│ 409 Conflict (duplicate) │ Return stored result │
│<───────────────────────────│ │
Key Takeaway
Retrying failed requests is essential in microservices because transient failures are common. A brief network glitch, a service restart, or a momentary CPU spike can cause a request to fail even though the service is healthy seconds later. Exponential backoff with jitter ensures that retries do not overwhelm the recovering service.
The critical insight is that retries are safe only when paired with idempotency. If the original request was processed but the response was lost, a retry will process the same request again. In a banking context, this means a customer is charged twice. Idempotency keys solve this: the client generates a unique key for each payment, and the server checks this key in a database before processing. If the key exists, the server returns the stored result without executing the payment again.
In our National Bank of India payment project, Rajesh sent a payment of 75,000 rupees from SAV2001 to Priya's SAV2002. The fraud-check-service was temporarily down, so the payment-service retried with exponential backoff: first after 1 second, then after 2 seconds. On the third attempt, fraud-check-service responded with CLEAN, and the payment was approved. The idempotency key PAY-001 was stored in the same database transaction as the payment record. When the client retried the same request (because the response was slow), the server returned HTTP 409 Conflict without processing the payment a second time.
At the senior level, remember to limit retry attempts to 2-3, use jitter to prevent thundering herds, and only retry on transient errors (5xx, timeout), never on client errors (4xx). At the principal level, monitor retry amplification across service chains and set global retry budgets.
Remember one sentence: every retry must be paired with an idempotency key, because after a timeout, you do not know if the server processed the original request or not.
---