Definition
- Two Generals Problem – a foundational impossibility result: two parties over an unreliable channel can never achieve guaranteed common-knowledge agreement because the last ack is always subject to loss.
- Ambiguous failure – a missing response is genuinely ambiguous: the caller cannot tell if the receiver acted or not.
- Retries – convert maybe-lost into at-least-once delivery, the achievable guarantee.
- Idempotency – keyed by a unique reference so a repeated operation is recognized and applied only once.
- Reconciliation – a periodic comparison of both sides that repairs any divergence.
- Eventual consistency – the sides may disagree briefly but converge to the same correct state.
Imagine National Bank of India must coordinate a settlement with a partner bank by sending messengers across a valley where some messengers get lost. Our bank sends "I will debit Rajesh and settle now: confirm." The partner replies "Confirmed." But did that confirmation arrive? There is always one more acknowledgement that could be lost, so neither bank can ever be certain the other acted. Instead of chasing perfect certainty, the banks keep retrying with a unique payment reference, make each step safe to repeat, and run an end-of-day reconciliation to catch any mismatch.
Understanding the Concepts
What is the Two Generals Problem?
Two generals on opposite hills must attack at the same time to win. They communicate only by messengers who can be captured. General A sends "attack at dawn." For A to be safe, A needs B's confirmation. But B cannot be safe until B knows A received B's ack. No finite exchange of messages ever makes both sides certain, because the last message can always be lost.
package com.bank.distributed;
public class TwoGeneralsProblem {
public static void main(String[] args) {
// Simulating the impossibility
System.out.println("General A sends: 'Attack at dawn'");
System.out.println("General B receives it and sends: 'Confirmed'");
System.out.println("Did A receive the confirmation? Unknown.");
System.out.println("A sends: 'I got your confirmation'");
System.out.println("Did B receive that? Unknown.");
System.out.println("...");
System.out.println("No finite chain of acks produces certainty.");
System.out.println("The last message can ALWAYS be lost.");
}
}
General A sends: 'Attack at dawn'
General B receives it and sends: 'Confirmed'
Did A receive the confirmation? Unknown.
A sends: 'I got your confirmation'
Did B receive that? Unknown.
...
No finite chain of acks produces certainty.
The last message can ALWAYS be lost.
How does this map to distributed systems?
Any two services communicating over a network face exactly this. The network can drop, delay, duplicate, or reorder messages. A request-and-response where the caller sends "do X" and waits for "done" has an irreducible ambiguity: if the response is lost, the caller cannot tell whether the receiver performed X or not.
package com.bank.distributed;
import java.math.BigDecimal;
public class AmbiguousFailure {
public static void main(String[] args) {
String paymentRef = "PAY-2024-001";
BigDecimal amount = new BigDecimal("50000.00");
System.out.println("Sender: 'Settle " + paymentRef +
" for " + amount + "'");
System.out.println("--- network timeout ---");
System.out.println();
System.out.println("Question: Did the partner bank settle it?");
System.out.println("Case A: Partner settled, response was lost");
System.out.println("Case B: Partner never received the request");
System.out.println("The sender CANNOT distinguish A from B!");
}
}
Sender: 'Settle PAY-2024-001 for 50000.00'
--- network timeout ---
Question: Did the partner bank settle it?
Case A: Partner settled, response was lost
Case B: Partner never received the request
The sender CANNOT distinguish A from B!
How do retries solve the delivery problem?
Retries convert maybe-lost into at-least-once delivery. On timeout, the sender resends. This is the only safe response to a lost ack: not retrying risks a lost payment, which is also unacceptable.
package com.bank.settlement;
import java.math.BigDecimal;
public class RetryingSettlementClient {
private final PartnerBankClient partner;
private static final int MAX_RETRIES = 3;
private static final long BACKOFF_MS = 1000;
public RetryingSettlementClient(PartnerBankClient partner) {
this.partner = partner;
}
public SettlementResult settle(String paymentRef,
BigDecimal amount) {
for (int attempt = 1; attempt <= MAX_RETRIES; attempt++) {
try {
return partner.settle(paymentRef, amount);
} catch (TimeoutException e) {
System.out.println("Attempt " + attempt +
" timed out, retrying...");
sleep(BACKOFF_MS * attempt);
}
}
throw new SettlementException(
"Settlement " + paymentRef + " failed after " +
MAX_RETRIES + " attempts");
}
private void sleep(long ms) {
try { Thread.sleep(ms); }
catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
Retry timeline:
Attempt 1: timeout (partner may or may not have settled)
Attempt 2: timeout (duplicate possible if attempt 1 succeeded)
Attempt 3: success (but was the settlement applied 1x, 2x, or 3x?)
Problem: retries cause duplicates -> need idempotency
How does idempotency make retries safe?
Because retries cause duplicates, every operation must be safe to apply more than once. The partner bank recognizes the payment reference and applies it only once, giving the effect of exactly-once even though delivery is at-least-once.
package com.bank.settlement;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class IdempotentSettlementHandler {
private final SettlementRepository repo;
private final AccountRepository accountRepo;
public IdempotentSettlementHandler(SettlementRepository repo,
AccountRepository accountRepo) {
this.repo = repo;
this.accountRepo = accountRepo;
}
@Transactional
public SettlementResult handle(String paymentRef,
BigDecimal amount,
String beneficiary) {
// Check if already processed
if (repo.existsByPaymentRef(paymentRef)) {
// Return the original result (idempotent)
return repo.findByPaymentRef(paymentRef).getResult();
}
// First time: process the settlement
accountRepo.credit(beneficiary, amount);
// Record the result atomically
repo.save(new Settlement(paymentRef, amount,
beneficiary, SettlementResult.SUCCESS));
return SettlementResult.SUCCESS;
}
}
Idempotency in action:
Request 1 (PAY-2024-001): process, credit Priya 50000, record
Request 2 (PAY-2024-001): already processed -> return original result
Request 3 (PAY-2024-001): already processed -> return original result
Result: Priya credited exactly once, regardless of retries
Why is reconciliation the safety net?
Reconciliation is a periodic process that compares both sides' records and repairs any divergence. It catches the rare case where an ack was lost and one side believes it acted while the other does not.
package com.bank.reconciliation;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
@Service
public class SettlementReconciliationService {
private final SettlementRepository ourRecords;
private final PartnerBankClient partner;
public SettlementReconciliationService(
SettlementRepository ourRecords,
PartnerBankClient partner) {
this.ourRecords = ourRecords;
this.partner = partner;
}
@Scheduled(cron = "0 0 23 * * *") // Every night at 11 PM
public void reconcile() {
List<Settlement> ours = ourRecords.findTodaysSettlements();
List<Settlement> theirs = partner.getTodaysSettlements();
Set<String> ourRefs = ours.stream()
.map(Settlement::getPaymentRef)
.collect(Collectors.toSet());
Set<String> theirRefs = theirs.stream()
.map(Settlement::getPaymentRef)
.collect(Collectors.toSet());
// Find mismatches
Set<String> missingAtPartner = ourRefs.stream()
.filter(ref -> !theirRefs.contains(ref))
.collect(Collectors.toSet());
Set<String> missingAtUs = theirRefs.stream()
.filter(ref -> !ourRefs.contains(ref))
.collect(Collectors.toSet());
if (!missingAtPartner.isEmpty() || !missingAtUs.isEmpty()) {
// Flag for manual investigation
System.out.println("Mismatches found: " +
missingAtPartner.size() + " missing at partner, " +
missingAtUs.size() + " missing at us");
}
}
}
Reconciliation report:
Our settlements today: 1,247
Partner's settlements today: 1,246
Missing at partner: 1 (PAY-2024-891)
Missing at us: 0
Action: re-send PAY-2024-891 or investigate
How do these four tools work together?
Acknowledgements with timeouts detect potential loss. Retries convert maybe-lost into at-least-once delivery. Idempotency makes duplicates harmless. Reconciliation catches anything that slipped through.
| Tool | What It Solves | Banking Example |
|---|---|---|
| Ack + Timeout | Detect potential message loss | Settlement response times out |
| Retry | Convert maybe-lost to at-least-once | Resend settlement instruction |
| Idempotency | Make duplicates harmless | Payment ref prevents double-credit |
| Reconciliation | Catch remaining mismatches | Nightly comparison of ledgers |
| Result | Eventual consistency | Both banks converge to same state |
What is the difference between exactly-once and effectively-once?
Exactly-once delivery is the impossible guarantee the Two Generals Problem forbids. Effectively-once is the achievable goal: at-least-once delivery combined with idempotent processing, so each operation's effect is applied exactly once even though the message may be delivered multiple times.
Exactly-once delivery: IMPOSSIBLE (last ack can always be lost)
At-least-once delivery: ACHIEVABLE (retry on timeout)
At-most-once delivery: ACHIEVABLE (never retry, accept loss)
Effectively-once = at-least-once + idempotency
Delivery: message may arrive 1, 2, or 3 times
Effect: business logic applied exactly once
How: idempotency key recognized on duplicate
What should a senior engineer know?
Never design a cross-service financial operation as if the network were reliable. Design every call to be retryable and every handler to be idempotent, and always have reconciliation.
package com.bank.settlement;
// Senior checklist for cross-service operations
public class DesignChecklist {
/*
* 1. Every request carries a stable idempotency key
* 2. Every handler checks the key before processing
* 3. The key and the business change commit atomically
* 4. Retries use exponential backoff with jitter
* 5. Reconciliation runs nightly and flags mismatches
* 6. Monitoring alerts on retry rate and reconciliation failures
* 7. Never assume the network is reliable
* 8. Never assume the other side committed
*/
}
Visual Overview
Two Generals Problem:
┌──────────┐ "attack" ┌──────────┐
│General A │ ──────────────> │General B │
│ │ <── "confirmed" │ │
│ │ (lost?) │ │
└──────────┘ └──────────┘
The last ack can always be lost.
Real-World Settlement:
┌──────────────┐ settle(PAY-001) ┌──────────────┐
│National Bank │ ─────────────────> │Partner Bank │
│of India │ <── ack (lost?) │ │
│ │ │ │
│ retry(PAY-001)──────────────────> │ idempotent: │
│ │ <── ack (received) │ same ref, │
│ │ │ no re-credit │
└──────┬───────┘ └──────┬───────┘
│ │
└──── nightly reconciliation ───────┘
compare payment refs
repair any mismatch
Key Takeaway
The Two Generals Problem is a foundational impossibility result that shapes how every distributed system must be designed. Two parties over an unreliable channel can never reach guaranteed agreement because the last acknowledgement can always be lost. This maps directly to banking: when National Bank of India sends a settlement instruction and the response is lost, the sender genuinely cannot tell whether the partner bank settled or not.
Because certainty is impossible, you design for the uncertainty with four tools. Retries convert maybe-lost into at-least-once delivery. Idempotency keyed by a unique payment reference makes those retries safe, so the partner bank recognizes a duplicate and returns the original result instead of double-crediting Priya. Reconciliation runs nightly, comparing both sides' payment references and flagging any mismatch for investigation. Together these produce eventual consistency: the two sides may disagree briefly but converge to the same correct state.
This is not a theoretical curiosity. Real inter-bank settlement works exactly this way: retries with unique transaction references, idempotent handlers that recognize duplicates, and nightly reconciliation. The practical lesson is to never design a cross-service financial operation as if the network were reliable. Design every call to be retryable, every handler to be idempotent, and always have reconciliation as the safety net.
Remember one sentence: you can never get guaranteed agreement over an unreliable channel because the last ack can always be lost, so build retryable calls, idempotent handlers, and reconciliation, and accept eventual consistency.
---