Definition
- The scenario – the server commits a payment, then crashes before the response reaches the client.
- Client sees – a timeout or connection error, indistinguishable from a genuine failure.
- Client must retry – not retrying risks a silently lost payment, which is also unacceptable.
- Double charge risk – the retry re-executes the payment, charging Rajesh twice.
- Server-side idempotency – the server records the idempotency key with the charge in one transaction, so a retry finds the record and returns the original result.
- Atomic commitment – the key and the charge must commit together; separate steps leave a crash window.
Rajesh hands the teller a payment slip, the teller processes it and the money moves, but before the receipt is handed back the power cuts out. Rajesh has no idea whether his payment went through. He comes back and submits the slip again. If the bank relied on Rajesh to "just not resubmit", it would be relying on something he cannot know. Only the bank, by recognizing the slip's unique number and seeing it was already processed, can safely hand back the original receipt instead of charging twice.
Understanding the Concepts
What is the exact failure sequence?
The request arrives, the server validates, performs the charge, and commits. The money has definitively moved. Then, before the HTTP response is written, the process crashes. The client sees only a timeout.
package com.bank.payment;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class PaymentService {
private final AccountRepository accountRepo;
public PaymentService(AccountRepository accountRepo) {
this.accountRepo = accountRepo;
}
@Transactional
public PaymentResult charge(String accountId, BigDecimal amount) {
accountRepo.debit(accountId, amount);
// Transaction COMMITS here
// Money has moved!
// *** CRASH HERE ***
// Process killed before response is sent
// Client sees: connection reset / timeout
return PaymentResult.success(); // Never reached
}
}
Failure timeline:
t=0ms: request arrives
t=10ms: debit committed (money moved)
t=11ms: JVM killed by autoscaler
t=2000ms: client times out
Client observes: "Connection timed out"
Client cannot tell: did the charge succeed or fail?
A lost-success and a genuine-failure look IDENTICAL.
Why must the client retry?
The client sees a timeout, which is ambiguous. Not retrying risks a silently lost payment. Retrying is the correct behavior, but it creates the double-charge risk.
Client decision:
Option A: Do not retry -> if payment failed, money is lost
Option B: Retry -> if payment succeeded, double charge
Both options are unsafe without server-side idempotency!
The client lacks information to decide correctly.
Client discipline CANNOT solve this.
How does server-side idempotency make the retry safe?
The client attaches a stable idempotency key. The server records the key with the charge in one transaction. On retry, the server finds the key and returns the stored result.
package com.bank.payment;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class IdempotentPaymentService {
private final AccountRepository accountRepo;
private final IdempotencyRepository idempotencyRepo;
public IdempotentPaymentService(AccountRepository accountRepo,
IdempotencyRepository idempotencyRepo) {
this.accountRepo = accountRepo;
this.idempotencyRepo = idempotencyRepo;
}
@Transactional
public PaymentResult charge(String idempotencyKey,
String accountId,
BigDecimal amount) {
// Check for existing result
IdempotencyRecord existing =
idempotencyRepo.findByKey(idempotencyKey);
if (existing != null) {
// Retry: return original result without charging again
return existing.getResult();
}
// First time: perform the charge
accountRepo.debit(accountId, amount);
// Record the key and result in SAME transaction
idempotencyRepo.save(new IdempotencyRecord(
idempotencyKey,
PaymentResult.success(accountId, amount)));
// If crash happens here, both charge AND key committed
// together, so retry will find the key.
return PaymentResult.success(accountId, amount);
}
}
First request (idempotencyKey = "PAY-KEY-001"):
Check key: not found
Debit Rajesh 50000
Save key + result
COMMIT (both in one transaction)
*** CRASH before response ***
Retry (same idempotencyKey = "PAY-KEY-001"):
Check key: found! (committed in same TX as charge)
Return stored result: success
Rajesh charged exactly once.
Why must the key and charge be in one transaction?
If you write the key in a separate step, a crash between them leaves a charge with no key, and the retry charges again. This is the dual-write problem.
package com.bank.payment;
import java.math.BigDecimal;
public class DualWriteProblem {
// WRONG: two separate writes
public void chargeWrong(String key, String accountId,
BigDecimal amount) {
accountRepo.debit(accountId, amount); // Step 1: committed
// *** CRASH HERE ***
idempotencyRepo.save(key, result); // Step 2: never reached!
// Retry: key not found -> charges AGAIN!
}
// CORRECT: one transaction
@Transactional
public void chargeCorrect(String key, String accountId,
BigDecimal amount) {
accountRepo.debit(accountId, amount); // Part of TX
idempotencyRepo.save(key, result); // Part of same TX
// COMMIT: both or neither
// Crash before commit: nothing happened, retry is safe
// Crash after commit: both saved, retry finds key
}
}
Dual-write problem:
Step 1 committed, Step 2 not -> charge without key -> double charge
Single transaction:
Both committed -> retry finds key -> safe
Neither committed -> retry processes from scratch -> safe
No in-between state possible!
How does the unique constraint handle concurrent retries?
The crash could happen mid-flight. The retry arrives before any record exists. A unique constraint on the key ensures only one wins.
CREATE TABLE idempotency_keys (
idempotency_key VARCHAR(64) PRIMARY KEY,
result JSONB NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
-- Unique constraint: INSERT fails for duplicate key
-- Only one concurrent request can win the INSERT
Concurrent retry race:
Request A: INSERT key='PAY-KEY-001' -> success (wins)
Request B: INSERT key='PAY-KEY-001' -> duplicate key error (loses)
Request B: read existing result -> return original
Result: exactly one charge, no matter how many concurrent retries
How does this apply to message consumers?
A consumer that commits the business change then crashes before acknowledging will see the message redelivered. The same idempotency pattern applies.
package com.bank.consumer;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class IdempotentPaymentConsumer {
private final PaymentRepository paymentRepo;
private final InboxRepository inbox;
public IdempotentPaymentConsumer(PaymentRepository paymentRepo,
InboxRepository inbox) {
this.paymentRepo = paymentRepo;
this.inbox = inbox;
}
@KafkaListener(topics = "payment-events")
@Transactional
public void onPayment(PaymentEvent event) {
// Same pattern: check inbox, process, record in same TX
if (inbox.existsByEventId(event.getEventId())) {
return; // Already processed
}
paymentRepo.applyPayment(event);
inbox.save(event.getEventId());
// Ack happens after commit
}
}
What are the common mistakes?
The most common mistake is writing the idempotency key in a separate transaction. Another is generating the key server-side per request instead of having the client send a stable key.
WRONG: server generates key per request
Request 1: key=UUID-111 -> charge
Retry: key=UUID-222 -> charge AGAIN (different key!)
CORRECT: client sends stable key
Request 1: key=PAY-KEY-001 -> charge
Retry: key=PAY-KEY-001 -> found, return original
What should a senior engineer know?
The response must be durably stored, not just the fact of success, so the retry can return the exact original result the client expected.
| Concern | Solution |
|---|---|
| Crash after commit | Key + charge in same TX |
| Concurrent retries | Unique constraint on key |
| Client retries | Stable idempotency key from client |
| Response replay | Store full response with key |
| Message consumers | Inbox table in same TX as business change |
| Key lifetime | TTL-based cleanup after max retry window |
Visual Overview
Normal Flow:
Client ──request──> Server ──commit──> DB
<──response── Server
Crash Scenario:
Client ──request──> Server ──commit──> DB
<── timeout *** CRASH ***
Client retries:
Client ──retry──> Server
├── Check idempotency key
├── Key found (committed with charge)
├── Return stored result
Client <──response── Server
Result: original result replayed, no double charge
Idempotency Record:
┌─────────────────────────────────────┐
│ Same Transaction │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Debit Rajesh │ │ Record Key + │ │
│ │ 50000.00 │ │ Result │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ COMMIT: both or neither │
└─────────────────────────────────────┘
Key Takeaway
When the server commits a payment and then crashes before the response reaches the client, the client sees only a timeout. A lost success and a genuine failure are completely indistinguishable to the client. The correct client behavior is to retry, because not retrying risks a silently lost payment. But retrying creates the double-charge risk. Client discipline cannot solve this because the client lacks the information to decide correctly.
The guarantee must live on the server with idempotency keyed to the logical operation. The client attaches a stable key that is the same across the original request and all retries. The server performs the charge and records the key with the response in the same transaction. A retry finds the committed key and replays the stored result. The reason this survives the crash is atomicity: if the charge committed, the key committed with it.
At National Bank of India, an autoscaler killed a payment pod right after the charge committed. Rajesh's app timed out and retried. Because the idempotency key and response were committed in the same transaction as the charge, the retry simply replayed the original receipt with no second charge. The unique constraint on the key also handled the case where two retries raced concurrently.
Remember one sentence: a crash after commit but before response is indistinguishable from failure, so the client must retry, and only server-side idempotency with the key and charge committed atomically can make that retry replay the original result rather than charge again.
---