Definition
- Request-level retry – Reactor's
retryWhen(Retry.backoff(...))re-issues failed interactions with exponential backoff, jitter, and a predicate filter. - Idempotency key – a client-generated unique reference sent with every retry so the server detects and ignores duplicates.
- Resumable session – a resume token negotiated at connect time lets a dropped transport reconnect and replay unacknowledged frames.
- Circuit breaker – Resilience4j counts failures and opens to fail fast with a fallback when a downstream is overwhelmed.
- Ordering – the circuit breaker wraps the retry, so retries happen while the breaker is closed and stop when it opens.
Imagine Rajesh's banking app talking to National Bank of India. If a single request hiccups, the app waits a moment and asks again, a little longer each time (client retry with backoff). If the phone line itself drops, the system automatically redials and resumes the exact conversation using a claim ticket (resumable session). And if the bank's desk is clearly overwhelmed and failing every call, the app stops hammering it and immediately shows a fallback for a while, occasionally testing whether the desk has recovered (circuit breaker).
Understanding the Concepts
How do request-level retries work?
On any interaction you append retryWhen(Retry.backoff(...)), which retries with exponentially increasing delays and built-in jitter to avoid synchronized retry storms.
package com.bank.rsocket;
import io.rsocket.RSocket;
import io.rsocket.util.DefaultPayload;
import reactor.core.publisher.Mono;
import reactor.util.retry.Retry;
import java.math.BigDecimal;
import java.time.Duration;
public class RequestRetryDemo {
public Mono<BigDecimal> getBalance(RSocket rSocket, String accountId) {
return rSocket.requestResponse(DefaultPayload.create(accountId))
.map(payload -> new BigDecimal(payload.getDataUtf8()))
.retryWhen(Retry.backoff(3, Duration.ofSeconds(1))
.jitter(0.5)
.filter(this::isTransient)
.doBeforeRetry(signal -> System.out.println(
"Retry #" + signal.totalRetries()
+ " for account " + accountId)))
.onErrorResume(ex -> {
System.out.println("All retries exhausted, fallback");
return Mono.just(new BigDecimal("150000.00"));
});
}
private boolean isTransient(Throwable ex) {
return ex instanceof java.net.ConnectException
|| ex instanceof java.util.concurrent.TimeoutException;
}
}
Retry timeline with backoff and jitter:
Attempt 0: immediate
Attempt 1: ~1.0s delay (+ jitter)
Attempt 2: ~2.0s delay (+ jitter)
Attempt 3: ~4.0s delay (+ jitter)
Exhausted: fallback
How do idempotency keys prevent double-debit?
For money-moving operations, every retry carries the same client-generated idempotency key so the server deduplicates.
package com.bank.rsocket;
import java.math.BigDecimal;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
public class IdempotentTransferService {
private final ConcurrentMap<String, BigDecimal> processed
= new ConcurrentHashMap<>();
public BigDecimal transfer(String idempotencyKey,
String from, String to,
BigDecimal amount) {
return processed.computeIfAbsent(idempotencyKey, key -> {
System.out.println("Processing transfer: " + key
+ " from " + from + " to " + to + " amount " + amount);
return amount;
});
}
public static void main(String[] args) {
IdempotentTransferService service = new IdempotentTransferService();
String key = UUID.randomUUID().toString();
// First call: processes the transfer
service.transfer(key, "SAV2001", "SAV2002",
new BigDecimal("50000.00"));
// Retry with same key: duplicate detected, Rajesh not double-debited
BigDecimal result = service.transfer(key, "SAV2001", "SAV2002",
new BigDecimal("50000.00"));
System.out.println("Result (no double-debit): " + result);
}
}
Processing transfer: a1b2c3d4 from SAV2001 to SAV2002 amount 50000.00
Result (no double-debit): 50000.00
How do resumable sessions differ from request retries?
Resumption recovers the whole connection and its in-flight streams transparently. A request retry re-issues one interaction.
| Mechanism | Scope | What it recovers | Application awareness |
|---|---|---|---|
| Request retry | Single interaction | Re-issues one request | Visible (retryWhen operator) |
| Session resumption | Entire connection | Replays unacknowledged frames | Transparent (streams continue) |
package com.bank.rsocket;
import io.rsocket.core.RSocketConnector;
import io.rsocket.core.Resume;
import io.rsocket.transport.netty.client.TcpClientTransport;
import reactor.util.retry.Retry;
import java.time.Duration;
public class ResumableVsRetry {
public static void main(String[] args) {
RSocketConnector.create()
// Session resumption: recovers the whole connection
.resume(new Resume()
.sessionDuration(Duration.ofMinutes(5))
.retry(Retry.backoff(10, Duration.ofSeconds(1))))
.connect(TcpClientTransport.create("localhost", 7000))
.flatMapMany(rSocket ->
rSocket.requestStream(
io.rsocket.util.DefaultPayload.create("RELIANCE"))
// Request retry: recovers individual requests
.retryWhen(Retry.backoff(3, Duration.ofSeconds(1))))
.subscribe(p -> System.out.println(
"Price: " + p.getDataUtf8()));
}
}
How does a circuit breaker protect against overload?
A circuit breaker counts failures and, once a threshold is crossed, opens and immediately fails subsequent calls with a fallback.
package com.bank.rsocket;
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.reactor.circuitbreaker.operator.CircuitBreakerOperator;
import io.rsocket.RSocket;
import io.rsocket.util.DefaultPayload;
import reactor.core.publisher.Mono;
import reactor.util.retry.Retry;
import java.math.BigDecimal;
import java.time.Duration;
public class CircuitBreakerDemo {
private final CircuitBreaker breaker = CircuitBreaker.of(
"balance-service",
CircuitBreakerConfig.custom()
.failureRateThreshold(50) // open at 50% failures
.waitDurationInOpenState(Duration.ofSeconds(30))
.slidingWindowSize(10)
.build());
public Mono<BigDecimal> getBalance(RSocket rSocket, String accountId) {
return rSocket.requestResponse(DefaultPayload.create(accountId))
.map(p -> new BigDecimal(p.getDataUtf8()))
// Retry inside the breaker
.retryWhen(Retry.backoff(2, Duration.ofSeconds(1))
.filter(this::isTransient))
// Breaker wraps the retry
.transformDeferred(CircuitBreakerOperator.of(breaker))
.onErrorResume(ex -> {
System.out.println("Breaker open or retries exhausted: "
+ ex.getMessage());
return Mono.just(new BigDecimal("150000.00"));
});
}
private boolean isTransient(Throwable ex) {
return !(ex instanceof IllegalArgumentException);
}
}
Circuit breaker states: CLOSED → normal operation, counting failures OPEN → fail fast, no calls to downstream HALF_OPEN → probe with a few trial calls ↓ If probes succeed → CLOSED (recovered) If probes fail → OPEN (still down)
Why does ordering matter between retry and circuit breaker?
The circuit breaker wraps the retry so retries happen inside a closed breaker but stop entirely when the breaker opens.
// CORRECT ordering: retry inside, breaker outside
rSocket.requestResponse(payload)
.retryWhen(Retry.backoff(3, Duration.ofSeconds(1))) // inner
.transformDeferred(CircuitBreakerOperator.of(breaker)) // outer
.onErrorResume(ex -> fallback());
// If breaker is CLOSED: retries happen normally
// If breaker is OPEN: fails immediately, no retries wasted
What about fire-and-forget and retries?
Fire-and-forget has no delivery guarantee and no response stream, so it has no meaningful retry semantics. Never use it for money movement.
// Fire-and-forget: best-effort, no retry possible
rSocket.fireAndForget(DefaultPayload.create("audit-login-event"));
// Use only for telemetry, logging, best-effort notifications
// For money movement: always use request-response with idempotency key
rSocket.requestResponse(DefaultPayload.create(
"{\"key\":\"" + idempotencyKey + "\",\"amount\":50000}"));
What are common mistakes with RSocket retry strategies?
The most common mistake is retrying non-idempotent requests like debitAccount without an idempotency key, risking double debits. Another error is configuring the circuit breaker after the retry policy in the reactive chain, so failed retries never trip the breaker. A third mistake is using fire-and-forget for operations that require confirmation, then being surprised when records are silently lost.
package com.bank.rsocket;
public class RSocketRetryMistakes {
public static void main(String[] args) {
System.out.println("WRONG: retry debit without idempotency key");
System.out.println(" .retryWhen(Retry.backoff(3, Duration.ofSeconds(1)))");
System.out.println(" // Each retry debits again -- Priya charged 4x!");
System.out.println();
System.out.println("CORRECT: include idempotency key in payload");
System.out.println(" DebitRequest.newBuilder()");
System.out.println(" .setIdempotencyKey(UUID.randomUUID().toString())");
System.out.println(" .setAmount(50000).build();");
System.out.println(" // Server deduplicates on idempotency key");
System.out.println();
System.out.println("WRONG: circuit breaker AFTER retry");
System.out.println(" .retryWhen(...).transform(circuitBreaker)");
System.out.println(" // Retries exhaust before breaker trips");
System.out.println();
System.out.println("CORRECT: circuit breaker BEFORE retry");
System.out.println(" .transform(circuitBreaker).retryWhen(...)");
}
}
Expected output:
WRONG: retry debit without idempotency key
.retryWhen(Retry.backoff(3, Duration.ofSeconds(1)))
// Each retry debits again -- Priya charged 4x!
CORRECT: include idempotency key in payload
DebitRequest.newBuilder()
.setIdempotencyKey(UUID.randomUUID().toString())
.setAmount(50000).build();
// Server deduplicates on idempotency key
WRONG: circuit breaker AFTER retry
.retryWhen(...).transform(circuitBreaker)
// Retries exhaust before breaker trips
CORRECT: circuit breaker BEFORE retry
.transform(circuitBreaker).retryWhen(...)
What should a senior engineer know?
- Always bound total retries and total time to avoid unbounded latency.
- Use jitter to prevent thundering herds when thousands of clients retry simultaneously.
- For request-channel sessions, decide whether to resume the channel or restart from a known checkpoint.
- In banking, queue transfers for asynchronous settlement when the breaker opens, so Rajesh sees "queued" rather than a hard failure.
// Bounded retry with total timeout
rSocket.requestResponse(payload)
.timeout(Duration.ofSeconds(10)) // total timeout
.retryWhen(Retry.backoff(3, Duration.ofSeconds(1))
.maxBackoff(Duration.ofSeconds(5))
.jitter(0.5));
Visual Overview
Three Layers of Failure Recovery:
Layer 1: Request Retry
┌─────────┐ fail ┌───────────┐ fail ┌───────────┐
│ Attempt │──────────▶│ Attempt 2 │──────────▶│ Attempt 3 │
│ 1 │ wait 1s │ (backoff) │ wait 2s │ (backoff) │
└─────────┘ └───────────┘ └───────────┘
│
▼ exhausted
┌───────────┐
│ Fallback │
└───────────┘
Layer 2: Session Resumption
┌────────┐ connected ┌────────┐
│ Client │────────────▶│ Server │
│ │ ← stream → │ │
│ │ │ │
│ │ TCP drops │ │
│ │─resume tok─▶│ │
│ │ ← stream → │ │ (continues)
└────────┘ └────────┘
Layer 3: Circuit Breaker
┌────────┐ ┌────────┐
│ Client │──┐ │ Server │
└────────┘ │ │(down) │
▼ └────────┘
┌──────────────────┐
│ Circuit Breaker │
│ CLOSED → OPEN │
│ (fail fast) │
│ → HALF_OPEN │
│ (probe) │
│ → CLOSED │
└──────────────────┘
Key Takeaway
RSocket handles retries and failure recovery in three layers, and the skill is knowing which layer handles which kind of failure. At the request level, use Reactor's retryWhen with exponential backoff and jitter, scoped by a predicate so you only retry transient errors like timeouts and connection resets, never validation errors. For anything that moves money, attach a client-generated idempotency key and reuse it across every retry, so if a retry duplicates a request that actually reached the server, the server dedupes it and Rajesh is never double-debited.
At the connection level, resumable sessions handle a dropped transport. When resume is enabled, the connection negotiates a token and tracks acknowledged frames, so a reconnect replays unacknowledged frames and in-flight streams continue transparently. That is different from a request retry: resumption recovers the whole connection, while a request retry re-issues a single interaction.
At the system level, a circuit breaker protects a failing downstream. When the failure rate crosses a threshold the breaker opens and calls fail fast with a fallback, which spares both the struggling service and your own threads. Order the breaker to wrap the retry so retries happen while the breaker is closed but stop entirely once it opens.
Remember one sentence: retryWhen with backoff and idempotency keys handles transient request failures, resumable sessions handle dropped connections, and circuit breakers fail fast to protect an overloaded peer.
---