Back to Resilience: Circuit Breaker, Retry, Bulkhead
Q9Question 9 of 9

How do you implement safe retries in microservices

Answer: Implement retries with exponential backoff and jitter using Spring Retry @Retryable, and pair every retry with a database-backed idempotency key to prevent duplicate payments when a response is lost and the client retries.

Key Concepts
  • A retry repeats a failed request hoping the failure was transient, but retries without idempotency cause duplicate side effects like double charges.
  • Exponential backoff increases the wait time between retries (1s, 2s, 4s) to give the failing service time to recover.
  • Jitter adds a random offset to the backoff delay to prevent thundering herd spikes when many clients retry simultaneously.
  • An idempotency key is a unique identifier sent by the client so the server can detect and reject duplicate requests.
  • The server stores the idempotency key and the response in the same database transaction to guarantee atomicity.
  • Only retry on transient errors (HTTP 5xx, timeouts), never on client errors (HTTP 4xx) which will always fail.
  • Spring Retry provides @Retryable with configurable backoff, max attempts, and @Recover for fallback when all retries fail.

Definition

Implement retries with exponential backoff and jitter using Spring Retry @Retryable, and pair every retry with a database-backed idempotency key to prevent duplicate payments when a response is lost and the client retries.
  • Retry is the pattern of repeating a failed network request because the failure might be transient (timeout, 503, connection reset) and the next attempt might succeed.
  • Exponential backoff is a retry delay strategy where the wait time doubles with each attempt (1s, 2s, 4s, 8s), giving the downstream service progressively more time to recover.
  • Jitter is a random offset added to the backoff delay so that multiple clients retrying simultaneously do not create thundering herd load spikes.
  • Idempotency key is a unique identifier (typically a UUID) generated by the client and sent with each request, allowing the server to detect and short-circuit duplicate requests.
  • @Retryable is a Spring Retry annotation that automatically retries a method when it throws a specified exception, with configurable max attempts and backoff parameters.

Think of Rajesh at the National Bank of India ATM. He inserts his card and requests a 50,000 rupee withdrawal. The ATM sends the request to the bank server, but the network is slow and times out. The ATM screen shows an error. Rajesh tries again with the same card. Without idempotency, the bank deducts 50,000 twice because each request looks new. With idempotency, the ATM generates a unique transaction ID for each withdrawal attempt. The bank checks: has this transaction ID been processed? Yes, so it returns the previous result without deducting again.

Understanding the Concepts

What is a retry in microservices?

A retry is when a client repeats a failed request to a service, hoping the failure was transient (temporary). Transient failures include network timeouts, TCP connection resets, HTTP 503 Service Unavailable, and brief database connection pool exhaustion. These failures resolve on their own within seconds.

The danger of retrying is that if the original request succeeded on the server but the response was lost, the retry will execute the operation a second time. For a payment service, this means charging the customer twice. This is why retries must always be paired with idempotency.

JAVA
package com.bank.retry;

public class RetryConceptDemo {

    sealed interface RetryDecision permits ShouldRetry, ShouldNotRetry {}
    record ShouldRetry(int httpStatus, String reason) implements RetryDecision {}
    record ShouldNotRetry(int httpStatus, String reason) implements RetryDecision {}

    static RetryDecision decideRetry(int httpStatus) {
        return switch (httpStatus) {
            case 408 -> new ShouldRetry(408, "Request Timeout: server was slow, retry");
            case 429 -> new ShouldRetry(429, "Too Many Requests: back off and retry");
            case 500 -> new ShouldRetry(500, "Internal Server Error: might be transient");
            case 502 -> new ShouldRetry(502, "Bad Gateway: upstream issue, retry");
            case 503 -> new ShouldRetry(503, "Service Unavailable: server overloaded, retry");
            case 400 -> new ShouldNotRetry(400, "Bad Request: fix the payload, do not retry");
            case 401 -> new ShouldNotRetry(401, "Unauthorized: fix credentials, do not retry");
            case 404 -> new ShouldNotRetry(404, "Not Found: resource does not exist, do not retry");
            case 409 -> new ShouldNotRetry(409, "Conflict: duplicate, do not retry");
            case 422 -> new ShouldNotRetry(422, "Unprocessable: business rule violation, do not retry");
            default -> new ShouldNotRetry(httpStatus, "Unknown status, do not retry");
        };
    }

    public static void main(String[] args) {
        var statuses = java.util.List.of(408, 429, 500, 503, 400, 404, 409, 422);
        for (var status : statuses) {
            var decision = decideRetry(status);
            String label = switch (decision) {
                case ShouldRetry r -> "RETRY    " + r.httpStatus() + " " + r.reason();
                case ShouldNotRetry n -> "NO RETRY " + n.httpStatus() + " " + n.reason();
            };
            System.out.println(label);
        }
    }
}
TEXT
Expected output:
RETRY    408 Request Timeout: server was slow, retry
RETRY    429 Too Many Requests: back off and retry
RETRY    500 Internal Server Error: might be transient
RETRY    503 Service Unavailable: server overloaded, retry
NO RETRY 400 Bad Request: fix the payload, do not retry
NO RETRY 404 Not Found: resource does not exist, do not retry
NO RETRY 409 Conflict: duplicate, do not retry
NO RETRY 422 Unprocessable: business rule violation, do not retry

What is exponential backoff with jitter?

Exponential backoff means increasing the wait time between retries exponentially: 1 second, 2 seconds, 4 seconds, 8 seconds, and so on. This gives the failing service time to recover.

Jitter adds a random component to the wait time. Without jitter, if 100 clients all started retrying at the same time (after a brief outage), they would all retry at the exact same intervals, creating thundering herd spikes that overload the recovering service again. With jitter, each client waits a slightly different amount of time, spreading the load.

The formula is: delay = min(baseDelay * 2^attempt + random(0, jitterRange), maxDelay)

JAVA
package com.bank.retry;

import java.util.Random;

public class ExponentialBackoffDemo {

    record RetryDelay(int attempt, long baseDelayMs, long jitterMs, long totalDelayMs) {}

    public static void main(String[] args) {
        long baseDelay = 1000;  // 1 second
        long maxDelay = 30000;  // 30 seconds cap
        var random = new Random(42);  // Fixed seed for reproducible output

        System.out.println("Exponential backoff with jitter:");
        System.out.println("Base delay: " + baseDelay + "ms, Max delay: " + maxDelay + "ms");
        System.out.println();

        for (int attempt = 1; attempt <= 5; attempt++) {
            long exponentialDelay = baseDelay * (long) Math.pow(2, attempt - 1);
            long jitter = random.nextLong(0, baseDelay);  // Jitter between 0 and baseDelay
            long totalDelay = Math.min(exponentialDelay + jitter, maxDelay);

            var delay = new RetryDelay(attempt, exponentialDelay, jitter, totalDelay);
            System.out.println("  Attempt " + delay.attempt()
                + ": base=" + delay.baseDelayMs() + "ms"
                + ", jitter=" + delay.jitterMs() + "ms"
                + ", total=" + delay.totalDelayMs() + "ms");
        }

        System.out.println();
        System.out.println("Without jitter: all 100 clients retry at 1s, 2s, 4s, 8s, 16s");
        System.out.println("  -> thundering herd at each retry interval");
        System.out.println("With jitter: client A retries at 1.3s, B at 1.7s, C at 0.9s...");
        System.out.println("  -> load is spread out, server recovers");
    }
}
TEXT
Expected output:
Exponential backoff with jitter:
Base delay: 1000ms, Max delay: 30000ms

  Attempt 1: base=1000ms, jitter=530ms, total=1530ms
  Attempt 2: base=2000ms, jitter=444ms, total=2444ms
  Attempt 3: base=4000ms, jitter=287ms, total=4287ms
  Attempt 4: base=8000ms, jitter=811ms, total=8811ms
  Attempt 5: base=16000ms, jitter=155ms, total=16155ms

Without jitter: all 100 clients retry at 1s, 2s, 4s, 8s, 16s
  -> thundering herd at each retry interval
With jitter: client A retries at 1.3s, B at 1.7s, C at 0.9s...
  -> load is spread out, server recovers

What is idempotency and why is it required for retries?

An operation is idempotent if executing it multiple times produces the same result as executing it once. For a payment debit of 50,000 rupees, executing the debit once should subtract 50,000. Executing the same debit request twice (because the client retried after a timeout) should still subtract only 50,000 total, not 100,000.

The standard approach is an idempotency key: a unique identifier generated by the client and sent with the request. The server stores this key in a database with a unique constraint. Before processing a request, the server checks if the key already exists. If it does, the request is a duplicate and the server returns the stored result without processing again.

ScenarioWithout IdempotencyWith Idempotency
First request succeedsDebit 50,000. Balance: 450,000Debit 50,000. Balance: 450,000. Store key.
Response lost, client retriesDebit 50,000 again. Balance: 400,000 (WRONG)Key exists, return stored result. Balance: 450,000 (CORRECT)
Client retries a third timeDebit 50,000 again. Balance: 350,000 (WRONG)Key exists, return stored result. Balance: 450,000 (CORRECT)
JAVA
package com.bank.retry;

import java.math.BigDecimal;
import java.util.HashMap;
import java.util.Map;

public class IdempotencyDemo {

    record PaymentResult(String idempotencyKey, BigDecimal amountDebited, BigDecimal newBalance, boolean wasDuplicate) {}

    static Map<String, PaymentResult> processedKeys = new HashMap<>();
    static BigDecimal balance = new BigDecimal("500000.00");

    static PaymentResult processPayment(String idempotencyKey, BigDecimal amount) {
        // Check if this request was already processed
        if (processedKeys.containsKey(idempotencyKey)) {
            var stored = processedKeys.get(idempotencyKey);
            System.out.println("  DUPLICATE detected for key=" + idempotencyKey + ", returning stored result");
            return new PaymentResult(idempotencyKey, stored.amountDebited(), stored.newBalance(), true);
        }

        // Process the payment
        balance = balance.subtract(amount);
        var result = new PaymentResult(idempotencyKey, amount, balance, false);
        processedKeys.put(idempotencyKey, result);
        System.out.println("  PROCESSED: key=" + idempotencyKey + ", debited=" + amount + ", newBalance=" + balance);
        return result;
    }

    public static void main(String[] args) {
        var key = "PAY-2024-001";
        var amount = new BigDecimal("50000.00");

        System.out.println("Balance before: " + balance);
        System.out.println();

        System.out.println("Attempt 1 (original request):");
        var r1 = processPayment(key, amount);
        System.out.println("  Result: debited=" + r1.amountDebited() + ", balance=" + r1.newBalance() + ", duplicate=" + r1.wasDuplicate());
        System.out.println();

        System.out.println("Attempt 2 (retry after timeout):");
        var r2 = processPayment(key, amount);
        System.out.println("  Result: debited=" + r2.amountDebited() + ", balance=" + r2.newBalance() + ", duplicate=" + r2.wasDuplicate());
        System.out.println();

        System.out.println("Attempt 3 (another retry):");
        var r3 = processPayment(key, amount);
        System.out.println("  Result: debited=" + r3.amountDebited() + ", balance=" + r3.newBalance() + ", duplicate=" + r3.wasDuplicate());
        System.out.println();

        System.out.println("Final balance: " + balance + " (debited only once, not three times)");
    }
}
TEXT
Expected output:
Balance before: 500000.00

Attempt 1 (original request):
  PROCESSED: key=PAY-2024-001, debited=50000.00, newBalance=450000.00
  Result: debited=50000.00, balance=450000.00, duplicate=false

Attempt 2 (retry after timeout):
  DUPLICATE detected for key=PAY-2024-001, returning stored result
  Result: debited=50000.00, balance=450000.00, duplicate=true

Attempt 3 (another retry):
  DUPLICATE detected for key=PAY-2024-001, returning stored result
  Result: debited=50000.00, balance=450000.00, duplicate=true

Final balance: 450000.00 (debited only once, not three times)

How is retry different from circuit breaker?

Retry and circuit breaker are complementary resilience patterns. Retry repeats a failed request hoping it succeeds on the next attempt. Circuit breaker stops sending requests to a service that is repeatedly failing, giving it time to recover.

FeatureRetryCircuit Breaker
PurposeHandle transient failuresPrevent cascading failures
When it activatesOn each individual failureAfter N consecutive failures
BehaviorSends the request againRejects requests immediately (fast-fail)
TimeoutAdds exponential backoff delayOpens for a fixed duration, then half-opens
RiskCan overload a struggling serviceCan reject requests even after service recovers
Works best withIdempotent operationsFallback responses or error pages
Spring libraryspring-retry @RetryableResilience4j @CircuitBreaker
JAVA
package com.bank.retry;

public class RetryVsCircuitBreaker {

    enum CircuitState { CLOSED, OPEN, HALF_OPEN }

    public static void main(String[] args) {
        System.out.println("=== Retry Pattern ===");
        System.out.println("Request 1: FAIL (503) -> wait 1s -> retry");
        System.out.println("Request 1: FAIL (503) -> wait 2s -> retry");
        System.out.println("Request 1: SUCCESS (200) -> done");
        System.out.println("Total time: ~3 seconds for 1 request");
        System.out.println();

        System.out.println("=== Circuit Breaker Pattern ===");
        System.out.println("State: CLOSED (normal operation)");
        System.out.println("Request 1: FAIL -> failure count=1");
        System.out.println("Request 2: FAIL -> failure count=2");
        System.out.println("Request 3: FAIL -> failure count=3 (threshold reached)");
        System.out.println("State: OPEN (reject all requests for 30 seconds)");
        System.out.println("Request 4: REJECTED immediately (no network call)");
        System.out.println("Request 5: REJECTED immediately");
        System.out.println("...30 seconds later...");
        System.out.println("State: HALF_OPEN (allow one test request)");
        System.out.println("Request 6: SUCCESS -> State: CLOSED (back to normal)");
        System.out.println();

        System.out.println("=== Best Practice: Use Both Together ===");
        System.out.println("Circuit Breaker wraps Retry:");
        System.out.println("  if (circuit is CLOSED) { try with retries } else { fast-fail }");
    }
}
TEXT
Expected output:
=== Retry Pattern ===
Request 1: FAIL (503) -> wait 1s -> retry
Request 1: FAIL (503) -> wait 2s -> retry
Request 1: SUCCESS (200) -> done
Total time: ~3 seconds for 1 request

=== Circuit Breaker Pattern ===
State: CLOSED (normal operation)
Request 1: FAIL -> failure count=1
Request 2: FAIL -> failure count=2
Request 3: FAIL -> failure count=3 (threshold reached)
State: OPEN (reject all requests for 30 seconds)
Request 4: REJECTED immediately (no network call)
Request 5: REJECTED immediately
...30 seconds later...
State: HALF_OPEN (allow one test request)
Request 6: SUCCESS -> State: CLOSED (back to normal)

=== Best Practice: Use Both Together ===
Circuit Breaker wraps Retry:
  if (circuit is CLOSED) { try with retries } else { fast-fail }

What is a complete banking project with retries and idempotency?

This project has a payment-service that processes payments. For each payment, it calls an external fraud-check-service via REST. The fraud-check-service is unreliable (simulated with WireMock in tests). The payment-service uses Spring Retry with exponential backoff and jitter, plus an idempotency key stored in a database to prevent duplicate payments.

Project structure

DIAGRAM
payment-retry-service/
├── pom.xml
├── docker-compose.yml
├── init-db.sql
└── src/
    ├── main/
    │   ├── java/com/bank/payment/
    │   │   ├── PaymentServiceApp.java
    │   │   ├── controller/
    │   │   │   └── PaymentController.java
    │   │   ├── service/
    │   │   │   └── PaymentService.java
    │   │   ├── client/
    │   │   │   └── FraudCheckClient.java
    │   │   ├── repository/
    │   │   │   ├── PaymentRepository.java
    │   │   │   └── IdempotencyRepository.java
    │   │   ├── model/
    │   │   │   ├── Payment.java
    │   │   │   └── IdempotencyRecord.java
    │   │   ├── dto/
    │   │   │   ├── PaymentRequest.java
    │   │   │   └── FraudCheckResponse.java
    │   │   ├── config/
    │   │   │   ├── RetryConfig.java
    │   │   │   └── RestClientConfig.java
    │   │   └── exception/
    │   │       ├── DuplicatePaymentException.java
    │   │       ├── FraudCheckFailedException.java
    │   │       └── GlobalExceptionHandler.java
    │   └── resources/
    │       └── application.yml
    └── test/
        └── java/com/bank/payment/
            └── PaymentRetryIntegrationTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><parentsyn-kw">>
        syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
        syn-kw"><artifactIdsyn-kw">>spring-boot-starter-parentsyn-kw"></artifactIdsyn-kw">>
        syn-kw"><versionsyn-kw">>3.4.1syn-kw"></versionsyn-kw">>
    syn-kw"></parentsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>payment-retry-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-websyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-data-jpasyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.retrysyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-retrysyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframeworksyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-aspectssyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.postgresqlsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>postgresqlsyn-kw"></artifactIdsyn-kw">>
            syn-kw"><scopesyn-kw">>runtimesyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-testsyn-kw"></artifactIdsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.wiremocksyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>wiremock-standalonesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>3.5.4syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

init-db.sql

SQL
CREATE TABLE IF NOT EXISTS payments (
    id                BIGSERIAL PRIMARY KEY,
    idempotency_key   VARCHAR(50) NOT NULL,
    from_account      VARCHAR(20) NOT NULL,
    to_account        VARCHAR(20) NOT NULL,
    amount            NUMERIC(15,2) NOT NULL,
    status            VARCHAR(20) NOT NULL DEFAULT 'PENDING',
    fraud_check_result VARCHAR(20),
    retry_count       INT NOT NULL DEFAULT 0,
    created_at        TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE IF NOT EXISTS idempotency_records (
    id                BIGSERIAL PRIMARY KEY,
    idempotency_key   VARCHAR(50) NOT NULL UNIQUE,
    response_status   VARCHAR(20) NOT NULL,
    response_body     TEXT,
    created_at        TIMESTAMP NOT NULL DEFAULT NOW()
);

Payment.java

JAVA
package com.bank.payment.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;

import java.math.BigDecimal;
import java.time.LocalDateTime;

@Entity
@Table(name = "payments")
public class Payment {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "idempotency_key", nullable = false)
    private String idempotencyKey;

    @Column(name = "from_account", nullable = false)
    private String fromAccount;

    @Column(name = "to_account", nullable = false)
    private String toAccount;

    @Column(nullable = false)
    private BigDecimal amount;

    @Column(nullable = false)
    private String status;

    @Column(name = "fraud_check_result")
    private String fraudCheckResult;

    @Column(name = "retry_count")
    private int retryCount;

    @Column(name = "created_at", nullable = false)
    private LocalDateTime createdAt = LocalDateTime.now();

    public Long getId() { return id; }
    public String getIdempotencyKey() { return idempotencyKey; }
    public void setIdempotencyKey(String key) { this.idempotencyKey = key; }
    public String getFromAccount() { return fromAccount; }
    public void setFromAccount(String fromAccount) { this.fromAccount = fromAccount; }
    public String getToAccount() { return toAccount; }
    public void setToAccount(String toAccount) { this.toAccount = toAccount; }
    public BigDecimal getAmount() { return amount; }
    public void setAmount(BigDecimal amount) { this.amount = amount; }
    public String getStatus() { return status; }
    public void setStatus(String status) { this.status = status; }
    public String getFraudCheckResult() { return fraudCheckResult; }
    public void setFraudCheckResult(String result) { this.fraudCheckResult = result; }
    public int getRetryCount() { return retryCount; }
    public void setRetryCount(int count) { this.retryCount = count; }
}

IdempotencyRecord.java

JAVA
package com.bank.payment.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;

import java.time.LocalDateTime;

@Entity
@Table(name = "idempotency_records")
public class IdempotencyRecord {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "idempotency_key", nullable = false, unique = true)
    private String idempotencyKey;

    @Column(name = "response_status", nullable = false)
    private String responseStatus;

    @Column(name = "response_body")
    private String responseBody;

    @Column(name = "created_at", nullable = false)
    private LocalDateTime createdAt = LocalDateTime.now();

    public Long getId() { return id; }
    public String getIdempotencyKey() { return idempotencyKey; }
    public void setIdempotencyKey(String key) { this.idempotencyKey = key; }
    public String getResponseStatus() { return responseStatus; }
    public void setResponseStatus(String status) { this.responseStatus = status; }
    public String getResponseBody() { return responseBody; }
    public void setResponseBody(String body) { this.responseBody = body; }
}

PaymentRequest.java and FraudCheckResponse.java

JAVA
package com.bank.payment.dto;

import java.math.BigDecimal;

public record PaymentRequest(
    String idempotencyKey,
    String fromAccount,
    String toAccount,
    BigDecimal amount,
    String description
) {}
JAVA
package com.bank.payment.dto;

public record FraudCheckResponse(String status, String riskLevel, String message) {}

PaymentRepository.java and IdempotencyRepository.java

JAVA
package com.bank.payment.repository;

import com.bank.payment.model.Payment;
import org.springframework.data.jpa.repository.JpaRepository;

public interface PaymentRepository extends JpaRepository<Payment, Long> {}
JAVA
package com.bank.payment.repository;

import com.bank.payment.model.IdempotencyRecord;
import org.springframework.data.jpa.repository.JpaRepository;

import java.util.Optional;

public interface IdempotencyRepository extends JpaRepository<IdempotencyRecord, Long> {
    Optional<IdempotencyRecord> findByIdempotencyKey(String key);
}

RetryConfig.java

JAVA
package com.bank.payment.config;

import org.springframework.context.annotation.Configuration;
import org.springframework.retry.annotation.EnableRetry;

@Configuration
@EnableRetry
public class RetryConfig {
    // @EnableRetry activates @Retryable annotations across the application.
    // Retry parameters are configured on each @Retryable method.
}

RestClientConfig.java

JAVA
package com.bank.payment.config;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestClient;

@Configuration
public class RestClientConfig {

    @Value("\${fraud.check.service.url:http://localhost:8090}")
    private String fraudCheckUrl;

    @Bean
    public RestClient fraudCheckRestClient() {
        var factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(2000);
        factory.setReadTimeout(3000);
        return RestClient.builder()
            .baseUrl(fraudCheckUrl)
            .requestFactory(factory)
            .build();
    }
}

FraudCheckClient.java (with @Retryable)

JAVA
package com.bank.payment.client;

import com.bank.payment.dto.FraudCheckResponse;
import com.bank.payment.exception.FraudCheckFailedException;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Recover;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import org.springframework.web.client.RestClientException;

import java.math.BigDecimal;
import java.util.Map;
import java.util.logging.Logger;

@Component
public class FraudCheckClient {

    private static final Logger LOG = Logger.getLogger(FraudCheckClient.class.getName());
    private final RestClient restClient;

    public FraudCheckClient(RestClient fraudCheckRestClient) {
        this.restClient = fraudCheckRestClient;
    }

    @Retryable(
        retryFor = RestClientException.class,
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2.0, random = true)
    )
    public FraudCheckResponse checkFraud(String accountId, BigDecimal amount) {
        LOG.info("Calling fraud-check-service for account=" + accountId + ", amount=" + amount);
        try {
            var body = Map.of("accountId", accountId, "amount", amount);
            return restClient.post()
                .uri("/fraud/check")
                .body(body)
                .retrieve()
                .body(FraudCheckResponse.class);
        } catch (RestClientException ex) {
            LOG.warning("Fraud check failed: " + ex.getMessage() + ". Will retry if attempts remain.");
            throw ex;
        }
    }

    @Recover
    public FraudCheckResponse recoverFraudCheck(RestClientException ex, String accountId, BigDecimal amount) {
        LOG.severe("All retry attempts exhausted for fraud check. account=" + accountId);
        throw new FraudCheckFailedException("Fraud check failed after all retries for " + accountId, ex);
    }
}

PaymentService.java (with idempotency check)

JAVA
package com.bank.payment.service;

import com.bank.payment.client.FraudCheckClient;
import com.bank.payment.dto.FraudCheckResponse;
import com.bank.payment.exception.DuplicatePaymentException;
import com.bank.payment.model.IdempotencyRecord;
import com.bank.payment.model.Payment;
import com.bank.payment.repository.IdempotencyRepository;
import com.bank.payment.repository.PaymentRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.math.BigDecimal;
import java.util.logging.Logger;

@Service
public class PaymentService {

    private static final Logger LOG = Logger.getLogger(PaymentService.class.getName());
    private final PaymentRepository paymentRepository;
    private final IdempotencyRepository idempotencyRepository;
    private final FraudCheckClient fraudCheckClient;

    public PaymentService(PaymentRepository paymentRepository,
                          IdempotencyRepository idempotencyRepository,
                          FraudCheckClient fraudCheckClient) {
        this.paymentRepository = paymentRepository;
        this.idempotencyRepository = idempotencyRepository;
        this.fraudCheckClient = fraudCheckClient;
    }

    @Transactional
    public Payment processPayment(String idempotencyKey, String from, String to, BigDecimal amount) {
        // Step 1: Idempotency check
        var existing = idempotencyRepository.findByIdempotencyKey(idempotencyKey);
        if (existing.isPresent()) {
            LOG.info("Duplicate payment detected: key=" + idempotencyKey);
            throw new DuplicatePaymentException(idempotencyKey);
        }

        // Step 2: Create payment record
        var payment = new Payment();
        payment.setIdempotencyKey(idempotencyKey);
        payment.setFromAccount(from);
        payment.setToAccount(to);
        payment.setAmount(amount);
        payment.setStatus("PENDING");
        paymentRepository.save(payment);

        // Step 3: Fraud check (with automatic retries via @Retryable)
        FraudCheckResponse fraudResult = fraudCheckClient.checkFraud(from, amount);
        payment.setFraudCheckResult(fraudResult.status());

        if ("REJECTED".equals(fraudResult.status())) {
            payment.setStatus("FRAUD_REJECTED");
            paymentRepository.save(payment);
        } else {
            payment.setStatus("APPROVED");
            paymentRepository.save(payment);
        }

        // Step 4: Record idempotency key
        var record = new IdempotencyRecord();
        record.setIdempotencyKey(idempotencyKey);
        record.setResponseStatus(payment.getStatus());
        record.setResponseBody(payment.getId().toString());
        idempotencyRepository.save(record);

        LOG.info("Payment processed: key=" + idempotencyKey + ", status=" + payment.getStatus());
        return payment;
    }
}

PaymentController.java

JAVA
package com.bank.payment.controller;

import com.bank.payment.dto.PaymentRequest;
import com.bank.payment.model.Payment;
import com.bank.payment.service.PaymentService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/payments")
public class PaymentController {

    private final PaymentService paymentService;

    public PaymentController(PaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @PostMapping
    public ResponseEntity<Payment> createPayment(@RequestBody PaymentRequest request) {
        var payment = paymentService.processPayment(
            request.idempotencyKey(), request.fromAccount(),
            request.toAccount(), request.amount());
        return ResponseEntity.ok(payment);
    }
}

Exception classes

JAVA
package com.bank.payment.exception;

public class DuplicatePaymentException extends RuntimeException {
    public DuplicatePaymentException(String idempotencyKey) {
        super("Duplicate payment: idempotency key " + idempotencyKey + " already processed");
    }
}
JAVA
package com.bank.payment.exception;

public class FraudCheckFailedException extends RuntimeException {
    public FraudCheckFailedException(String message, Throwable cause) {
        super(message, cause);
    }
}

GlobalExceptionHandler.java

JAVA
package com.bank.payment.exception;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;

import java.util.Map;

@RestControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(DuplicatePaymentException.class)
    public ResponseEntity<Map<String, String>> handleDuplicate(DuplicatePaymentException ex) {
        return ResponseEntity.status(HttpStatus.CONFLICT)
            .body(Map.of("error", ex.getMessage()));
    }

    @ExceptionHandler(FraudCheckFailedException.class)
    public ResponseEntity<Map<String, String>> handleFraudCheckFailed(FraudCheckFailedException ex) {
        return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .body(Map.of("error", ex.getMessage()));
    }
}

application.yml

YAML
server:
  port: 8083

spring:
  application:
    name: payment-retry-service
  datasource:
    url: jdbc:postgresql://localhost:5432/bank
    username: bank
    password: bank123
  jpa:
    hibernate:
      ddl-auto: none

fraud:
  check:
    service:
      url: http://localhost:8090

How to run

BASH
# Step 1: Start PostgreSQL and initialize tables
docker-compose up -d postgres

# Step 2: Build the project
mvn clean package -DskipTests

# Step 3: Start the application
java -jar target/payment-retry-service-1.0.0.jar

# Step 4: Send a payment request
curl -s -X POST http://localhost:8083/payments \
  -H "Content-Type: application/json" \
  -d '{"idempotencyKey":"PAY-001","fromAccount":"SAV2001","toAccount":"SAV2002","amount":75000,"description":"rent"}'

# Step 5: Send the SAME request again (duplicate)
curl -s -X POST http://localhost:8083/payments \
  -H "Content-Type: application/json" \
  -d '{"idempotencyKey":"PAY-001","fromAccount":"SAV2001","toAccount":"SAV2002","amount":75000,"description":"rent"}'
TEXT
Expected output (first request):
{"id":1,"idempotencyKey":"PAY-001","fromAccount":"SAV2001","toAccount":"SAV2002",
 "amount":75000.00,"status":"APPROVED","fraudCheckResult":"CLEAN","retryCount":0}

Expected output (duplicate request):
HTTP 409 Conflict
{"error":"Duplicate payment: idempotency key PAY-001 already processed"}

What are common mistakes with retries?

Wrong: retry without idempotency causes double charge

JAVA
package com.bank.payment.wrong;

import java.math.BigDecimal;

public class WrongRetryWithoutIdempotency {
    static BigDecimal balance = new BigDecimal("500000.00");

    static void processPayment(BigDecimal amount) {
        balance = balance.subtract(amount);
        System.out.println("  Debited " + amount + ", new balance: " + balance);
    }

    public static void main(String[] args) {
        var amount = new BigDecimal("50000.00");
        System.out.println("WRONG: Retry without idempotency");
        System.out.println("Balance before: " + balance);

        System.out.println("Attempt 1: server processes, but response is lost");
        processPayment(amount);

        System.out.println("Attempt 2: client retries, server processes AGAIN");
        processPayment(amount);

        System.out.println("Attempt 3: client retries again, server processes AGAIN");
        processPayment(amount);

        System.out.println("Final balance: " + balance);
        System.out.println("Expected: 450000.00, Actual: " + balance + " (CUSTOMER CHARGED 3x!)");
    }
}
TEXT
Expected output:
WRONG: Retry without idempotency
Balance before: 500000.00
Attempt 1: server processes, but response is lost
  Debited 50000.00, new balance: 450000.00
Attempt 2: client retries, server processes AGAIN
  Debited 50000.00, new balance: 400000.00
Attempt 3: client retries again, server processes AGAIN
  Debited 50000.00, new balance: 350000.00
Final balance: 350000.00
Expected: 450000.00, Actual: 350000.00 (CUSTOMER CHARGED 3x!)

Correct: retry with idempotency key prevents duplicate charges

JAVA
package com.bank.payment.correct;

import java.math.BigDecimal;
import java.util.HashSet;
import java.util.Set;

public class CorrectRetryWithIdempotency {
    static BigDecimal balance = new BigDecimal("500000.00");
    static Set<String> processedKeys = new HashSet<>();

    static boolean processPayment(String idempotencyKey, BigDecimal amount) {
        if (processedKeys.contains(idempotencyKey)) {
            System.out.println("  DUPLICATE key=" + idempotencyKey + ", returning stored result");
            return true;  // Return success without processing again
        }
        balance = balance.subtract(amount);
        processedKeys.add(idempotencyKey);
        System.out.println("  Debited " + amount + ", new balance: " + balance);
        return true;
    }

    public static void main(String[] args) {
        var amount = new BigDecimal("50000.00");
        System.out.println("CORRECT: Retry with idempotency key");
        System.out.println("Balance before: " + balance);

        System.out.println("Attempt 1: server processes, response is lost");
        processPayment("PAY-001", amount);

        System.out.println("Attempt 2: client retries with same key");
        processPayment("PAY-001", amount);

        System.out.println("Attempt 3: client retries again with same key");
        processPayment("PAY-001", amount);

        System.out.println("Final balance: " + balance + " (debited only ONCE)");
    }
}
TEXT
Expected output:
CORRECT: Retry with idempotency key
Balance before: 500000.00
Attempt 1: server processes, response is lost
  Debited 50000.00, new balance: 450000.00
Attempt 2: client retries with same key
  DUPLICATE key=PAY-001, returning stored result
Attempt 3: client retries again with same key
  DUPLICATE key=PAY-001, returning stored result
Final balance: 450000.00 (debited only ONCE)

How do you test retries with WireMock?

WireMock can simulate transient failures by returning different responses for consecutive calls to the same endpoint. The first two calls return HTTP 503 (Service Unavailable), and the third call returns HTTP 200 with a success body. This tests that your retry logic works correctly.

JAVA
package com.bank.payment;

import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.stubbing.Scenario;

import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
import static com.github.tomakehurst.wiremock.client.WireMock.post;
import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;

public class RetryWireMockTest {
    public static void main(String[] args) {
        var wireMock = new WireMockServer(8090);
        wireMock.start();

        // First call: return 503
        wireMock.stubFor(post(urlEqualTo("/fraud/check"))
            .inScenario("retry-test")
            .whenScenarioStateIs(Scenario.STARTED)
            .willReturn(aResponse().withStatus(503).withBody("Service Unavailable"))
            .willSetStateTo("ATTEMPT_2"));

        // Second call: return 503
        wireMock.stubFor(post(urlEqualTo("/fraud/check"))
            .inScenario("retry-test")
            .whenScenarioStateIs("ATTEMPT_2")
            .willReturn(aResponse().withStatus(503).withBody("Service Unavailable"))
            .willSetStateTo("ATTEMPT_3"));

        // Third call: return 200
        wireMock.stubFor(post(urlEqualTo("/fraud/check"))
            .inScenario("retry-test")
            .whenScenarioStateIs("ATTEMPT_3")
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"status\":\"CLEAN\",\"riskLevel\":\"LOW\",\"message\":\"No fraud detected\"}")));

        System.out.println("WireMock started on port 8090");
        System.out.println("Scenario: 2 failures then 1 success");
        System.out.println("  Call 1: HTTP 503 (triggers retry)");
        System.out.println("  Call 2: HTTP 503 (triggers retry)");
        System.out.println("  Call 3: HTTP 200 {status:CLEAN}");
        System.out.println();
        System.out.println("With @Retryable(maxAttempts=3, backoff=@Backoff(delay=1000, multiplier=2)):");
        System.out.println("  Attempt 1: FAIL (503), wait ~1s");
        System.out.println("  Attempt 2: FAIL (503), wait ~2s");
        System.out.println("  Attempt 3: SUCCESS (200), fraud check CLEAN");
        System.out.println("  Total time: ~3 seconds");

        wireMock.stop();
    }
}
TEXT
Expected output:
WireMock started on port 8090
Scenario: 2 failures then 1 success
  Call 1: HTTP 503 (triggers retry)
  Call 2: HTTP 503 (triggers retry)
  Call 3: HTTP 200 {status:CLEAN}

With @Retryable(maxAttempts=3, backoff=@Backoff(delay=1000, multiplier=2)):
  Attempt 1: FAIL (503), wait ~1s
  Attempt 2: FAIL (503), wait ~2s
  Attempt 3: SUCCESS (200), fraud check CLEAN
  Total time: ~3 seconds

What should a senior or principal engineer know?

At the senior level, you must understand the tradeoffs of retry strategies:

  • Retry budget: limit the total number of retries per second across all clients. If you have 100 instances each retrying 3 times, you send 300 requests to a struggling service. A global retry budget caps the total.
  • Retry amplification: in a chain of services (A calls B calls C), if each service retries 3 times, a single failure in C can generate 3 x 3 = 9 requests. Keep retry counts low (2-3 max) and only retry at the outermost caller.
  • Idempotency key TTL: idempotency records should expire. A key from 30 days ago should not block a new request. Use a TTL of 24-48 hours in the database.
  • Idempotency scope: the key should be scoped to the operation, not just the request. If the same amount from the same account with the same description is sent twice, it could be two legitimate payments. The client must generate a unique key for each intended payment.
JAVA
package com.bank.payment;

public class SeniorRetryKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal retry and idempotency checklist:");
        System.out.println("  1. Always pair retries with idempotency keys");
        System.out.println("  2. Use exponential backoff + jitter, never fixed delay");
        System.out.println("  3. Only retry on transient errors (5xx, timeout), not 4xx");
        System.out.println("  4. Max 2-3 retry attempts per call");
        System.out.println("  5. Add retry budget to prevent retry storms");
        System.out.println("  6. Beware retry amplification in service chains");
        System.out.println("  7. Set TTL on idempotency keys (24-48 hours)");
        System.out.println("  8. Client generates the idempotency key, not the server");
        System.out.println("  9. Store idempotency key + result in the same transaction");
        System.out.println("  10. Log every retry attempt with correlation ID for debugging");
    }
}
TEXT
Expected output:
Senior/Principal retry and idempotency checklist:
  1. Always pair retries with idempotency keys
  2. Use exponential backoff + jitter, never fixed delay
  3. Only retry on transient errors (5xx, timeout), not 4xx
  4. Max 2-3 retry attempts per call
  5. Add retry budget to prevent retry storms
  6. Beware retry amplification in service chains
  7. Set TTL on idempotency keys (24-48 hours)
  8. Client generates the idempotency key, not the server
  9. Store idempotency key + result in the same transaction
  10. Log every retry attempt with correlation ID for debugging

What are common interview follow-up questions?

  • What happens if the idempotency key table is in a different database than the payment table? (You lose atomicity. If the payment is saved but the idempotency record is not, a retry will process the payment again. Always store both in the same database and the same transaction.)
  • Should you retry on HTTP 400 Bad Request? (No. A 400 means the request itself is invalid. Retrying the same invalid request will always fail. Fix the request, do not retry.)
  • What is the difference between @Retryable and Resilience4j Retry? (@Retryable is from Spring Retry and uses AOP proxies. Resilience4j is a standalone library with decorators. Resilience4j offers more features like retry rate limiting and better metrics integration.)
  • How do you handle a retry that exceeds the timeout budget? (Set a total timeout that includes all retry delays. If the total budget is 10 seconds and the first two attempts used 6 seconds, the third attempt only has 4 seconds left.)
  • Can you use idempotency with async messaging? (Yes. The message ID or a custom idempotency key in the message header serves the same purpose. The consumer checks the key before processing.)

Visual Overview

DIAGRAM
 Retry with Exponential Backoff and Idempotency Flow
 =====================================================

 Client                    payment-service              fraud-check-service
   │                            │                              │
   │  POST /payments            │                              │
   │  Idempotency-Key: PAY-001  │                              │
   │───────────────────────────>│                              │
   │                            │                              │
   │                            │  Check idempotency table     │
   │                            │  Key PAY-001 not found       │
   │                            │                              │
   │                            │  POST /fraud/check           │
   │                            │─────────────────────────────>│
   │                            │              503 Unavailable │
   │                            │<─────────────────────────────│
   │                            │                              │
   │                            │  Wait 1s (backoff + jitter)  │
   │                            │                              │
   │                            │  POST /fraud/check (retry 1) │
   │                            │─────────────────────────────>│
   │                            │              503 Unavailable │
   │                            │<─────────────────────────────│
   │                            │                              │
   │                            │  Wait 2s (backoff + jitter)  │
   │                            │                              │
   │                            │  POST /fraud/check (retry 2) │
   │                            │─────────────────────────────>│
   │                            │          200 OK {CLEAN}      │
   │                            │<─────────────────────────────│
   │                            │                              │
   │                            │  Save payment + idem key     │
   │                            │  (same DB transaction)       │
   │                            │                              │
   │  200 OK {APPROVED}         │                              │
   │<───────────────────────────│                              │
   │                            │                              │
   │  POST /payments (retry)    │                              │
   │  Idempotency-Key: PAY-001  │                              │
   │───────────────────────────>│                              │
   │                            │  Key PAY-001 FOUND           │
   │  409 Conflict (duplicate)  │  Return stored result        │
   │<───────────────────────────│                              │

Key Takeaway

Retrying failed requests is essential in microservices because transient failures are common. A brief network glitch, a service restart, or a momentary CPU spike can cause a request to fail even though the service is healthy seconds later. Exponential backoff with jitter ensures that retries do not overwhelm the recovering service.

The critical insight is that retries are safe only when paired with idempotency. If the original request was processed but the response was lost, a retry will process the same request again. In a banking context, this means a customer is charged twice. Idempotency keys solve this: the client generates a unique key for each payment, and the server checks this key in a database before processing. If the key exists, the server returns the stored result without executing the payment again.

In our National Bank of India payment project, Rajesh sent a payment of 75,000 rupees from SAV2001 to Priya's SAV2002. The fraud-check-service was temporarily down, so the payment-service retried with exponential backoff: first after 1 second, then after 2 seconds. On the third attempt, fraud-check-service responded with CLEAN, and the payment was approved. The idempotency key PAY-001 was stored in the same database transaction as the payment record. When the client retried the same request (because the response was slow), the server returned HTTP 409 Conflict without processing the payment a second time.

At the senior level, remember to limit retry attempts to 2-3, use jitter to prevent thundering herds, and only retry on transient errors (5xx, timeout), never on client errors (4xx). At the principal level, monitor retry amplification across service chains and set global retry budgets.

Remember one sentence: every retry must be paired with an idempotency key, because after a timeout, you do not know if the server processed the original request or not.

---

Share
Use ← → arrow keys to navigate