Definition
- Why not 2PC – External banking partners expose REST or messaging APIs, not XA transactional resources, so you cannot enroll them in a two-phase commit, and even if you could, the blocking locks are catastrophic for payments.
- Bulkhead – Each partner gets its own queue and dedicated consumer so a down partner's backlog does not starve the healthy ones.
- Circuit breaker – Counts recent failures per partner and opens to fail fast, stopping threads from piling up on a dead endpoint.
- Idempotency – Every leg carries the transaction ID as an idempotency key; replays return the stored result instead of debiting again.
- Saga + reconciliation – Money moves leg by leg with compensating reversals, and periodic reconciliation compares ledgers to resolve any divergence.
Imagine Rajesh sends money through National Bank of India that must pass through three partner banks. His money leaves his account, reaches the first partner, and then the second partner's phone line goes dead. The bank gives each partner its own clerk and in-tray (bulkhead), watches each line and stops calling when one is dead (circuit breaker), stamps every leg with Rajesh's transaction ID so replays say "already done" (idempotency), and moves money leg by leg with undo steps if a leg fails (Saga). A nightly reconciliation checks everyone's books agree.
Understanding the Concepts
Why is two-phase commit the wrong answer?
Two-phase commit needs every participant to be an XA-capable transactional resource. External banking partners expose REST or messaging APIs, not XA resources, so you cannot enroll them. Even if you could, 2PC is blocking: prepared participants hold locks until the coordinator decides, and if the coordinator crashes, they are stuck in doubt.
package com.bank.payment.antipattern;
// WRONG: attempting 2PC across external REST partners
public class TwoPhaseCommitWrong {
// Partner A: REST API at https://partner-a.com/api/transfer
// Partner B: REST API at https://partner-b.com/api/transfer
// These are NOT XA resources. You cannot:
// 1. "prepare" them (they have no prepare endpoint)
// 2. "commit" or "rollback" them atomically
// 3. Hold locks across independent organizations
// If the coordinator dies after "prepare", both partners are stuck.
}
How does the per-partner bulkhead work?
Each partner gets its own queue and dedicated consumer pool, so a down partner's backlog does not starve the healthy ones.
package com.bank.payment.resilience;
import java.util.Map;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class PartnerBulkhead {
private final Map<String, ThreadPoolExecutor> partnerPools;
public PartnerBulkhead() {
partnerPools = Map.of(
"HDFC", createPool("HDFC"),
"ICICI", createPool("ICICI"),
"SBI", createPool("SBI")
);
}
private ThreadPoolExecutor createPool(String partner) {
return new ThreadPoolExecutor(
5, 10, 60, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(50) // bounded queue per partner
);
}
public void submitToPartner(String partner, Runnable task) {
ThreadPoolExecutor pool = partnerPools.get(partner);
if (pool == null) throw new IllegalArgumentException("Unknown partner: " + partner);
pool.submit(task); // If HDFC is down, only HDFC's queue fills up
}
}
How does the circuit breaker detect failure early?
Each partner has a breaker that counts recent failures. When they cross a threshold, the breaker opens and subsequent calls fail immediately instead of waiting on a doomed call.
package com.bank.payment.resilience;
import java.time.Instant;
import java.util.concurrent.atomic.AtomicInteger;
public class SimpleCircuitBreaker {
private enum State { CLOSED, OPEN, HALF_OPEN }
private State state = State.CLOSED;
private final AtomicInteger failureCount = new AtomicInteger(0);
private final int threshold = 5;
private final long cooldownMs = 30_000;
private Instant openedAt;
public boolean allowRequest() {
if (state == State.CLOSED) return true;
if (state == State.OPEN) {
if (Instant.now().toEpochMilli() - openedAt.toEpochMilli() > cooldownMs) {
state = State.HALF_OPEN;
return true; // trial request
}
return false; // fail fast
}
return true; // HALF_OPEN: allow one trial
}
public void recordSuccess() {
failureCount.set(0);
state = State.CLOSED;
}
public void recordFailure() {
if (failureCount.incrementAndGet() >= threshold) {
state = State.OPEN;
openedAt = Instant.now();
}
}
}
How does transaction-ID idempotency prevent double-debit?
Every leg carries the transaction ID. Each partner interaction records the result. On retry, the stored result is returned instead of debiting again.
package com.bank.payment.service;
import java.math.BigDecimal;
import java.util.concurrent.ConcurrentHashMap;
public class IdempotentPartnerClient {
private final ConcurrentHashMap<String, String> processedTransactions =
new ConcurrentHashMap<>();
public String transferToPartner(String transactionId, String partnerId,
BigDecimal amount) {
// Check if already processed
String existingResult = processedTransactions.get(transactionId);
if (existingResult != null) {
System.out.println("Idempotent hit: " + transactionId + " -> " + existingResult);
return existingResult; // No double-debit
}
// Execute the transfer
String result = callPartnerApi(partnerId, transactionId, amount);
processedTransactions.put(transactionId, result);
return result;
}
private String callPartnerApi(String partnerId, String txnId, BigDecimal amount) {
// REST call to partner
return "SUCCESS";
}
}
# First call: transferToPartner("TXN-001", "HDFC", 50000) -> executes, returns SUCCESS
# Retry call: transferToPartner("TXN-001", "HDFC", 50000) -> idempotent hit, returns SUCCESS
# Rajesh is never double-debited.
How does the Saga with compensations work?
The payment is modeled as a sequence of local steps with compensating reversals. Money is held in a durable in-flight escrow tied to the transaction ID.
package com.bank.payment.saga;
import java.math.BigDecimal;
public class CrossBorderPaymentSaga {
private final IdempotentPartnerClient partnerClient;
private final EscrowService escrowService;
public CrossBorderPaymentSaga(IdempotentPartnerClient partnerClient,
EscrowService escrowService) {
this.partnerClient = partnerClient;
this.escrowService = escrowService;
}
public void execute(String txnId, String senderId, BigDecimal amount) {
// Step 1: Debit sender into escrow (money is always accounted for)
escrowService.debitToEscrow(txnId, senderId, amount);
// Step 2: Transfer leg 1 (HDFC)
String leg1 = partnerClient.transferToPartner(txnId + "-L1", "HDFC", amount);
if (!"SUCCESS".equals(leg1)) {
// Compensation: refund sender from escrow
escrowService.refundFromEscrow(txnId, senderId, amount);
return;
}
// Step 3: Transfer leg 2 (ICICI)
String leg2 = partnerClient.transferToPartner(txnId + "-L2", "ICICI", amount);
if (!"SUCCESS".equals(leg2)) {
// Compensation: reverse leg 1 + refund sender
partnerClient.transferToPartner(txnId + "-L1-REV", "HDFC", amount.negate());
escrowService.refundFromEscrow(txnId, senderId, amount);
return;
}
// All legs complete: release escrow
escrowService.releaseEscrow(txnId);
}
}
How does reconciliation close the loop?
A periodic process compares your ledger against each partner's ledger by transaction ID and resolves discrepancies.
package com.bank.payment.reconciliation;
import java.util.List;
import java.util.Map;
public class ReconciliationJob {
public void reconcile(String partnerId) {
List<String> ourPending = getOurPendingTransactions(partnerId);
for (String txnId : ourPending) {
String partnerStatus = queryPartnerStatus(partnerId, txnId);
if ("COMPLETED".equals(partnerStatus)) {
markComplete(txnId);
} else if ("NOT_FOUND".equals(partnerStatus)) {
retryOrCompensate(txnId);
} else {
flagForManualReview(txnId);
}
}
}
private List<String> getOurPendingTransactions(String p) { return List.of(); }
private String queryPartnerStatus(String p, String t) { return "COMPLETED"; }
private void markComplete(String t) { }
private void retryOrCompensate(String t) { }
private void flagForManualReview(String t) { }
}
What are common mistakes in this scenario?
The most common mistake is attempting two-phase commit across external REST APIs. The second is not using idempotency keys, which leads to double-debits on retry. The third is not having a bulkhead, so one down partner blocks all partners.
package com.bank.payment.antipattern;
// WRONG: shared thread pool for all partners
public class SharedPoolWrong {
// If HDFC is down, all 50 threads block on HDFC calls.
// ICICI and SBI payments cannot proceed.
// One partner's outage becomes a total outage.
}
// CORRECT: per-partner bulkhead (see PartnerBulkhead above)
What are common interview follow-up questions?
How do you handle the case where a partner processed the request but the response was lost? How do you handle a compensation that itself fails? What is the difference between an escrow hold and a direct debit? How does reconciliation handle timezone differences between partners? How do you prioritize reconciliation items?
Visual Overview
┌───────────────────────────────────────────────────────────┐ │ CROSS-BORDER PAYMENT WITH BULKHEAD + SAGA │ ├───────────────────────────────────────────────────────────┤ │ │ │ Rajesh's Account │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ Debit to │ (money in durable escrow) │ │ │ Escrow │ │ │ └──────┬───────┘ │ │ │ │ │ ┌────▼────────────────────────────────────┐ │ │ │ Per-Partner Bulkhead Queues │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ │ │ HDFC Q │ │ ICICI Q │ │ SBI Q │ │ │ │ │ │ + pool │ │ + pool │ │ + pool │ │ │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ └───────┼────────────┼────────────┼───────┘ │ │ │ │ │ │ │ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ │ │ │ CB │ │ CB │ │ CB │ (circuit │ │ │HDFC │ │ICICI│ │ SBI │ breakers) │ │ └──┬──┘ └──┬──┘ └──┬──┘ │ │ │ │ │ │ │ Partner Partner Partner │ │ HDFC API ICICI API SBI API │ │ │ │ On failure: Saga compensates (refund escrow) │ │ On recovery: Retry with same txn ID (idempotent) │ │ Nightly: Reconciliation by txn ID │ └───────────────────────────────────────────────────────────┘
Key Takeaway
When a payment spans several external banking partners and one goes down after Rajesh's money has left his account, Pankaj guarantees no money is lost and no double-debit by composing four mechanisms. First, two-phase commit is the wrong tool because external partners are REST endpoints, not XA resources, and blocking locks are catastrophic for payments. Second, each partner gets its own bulkheaded queue and consumer pool so a down partner's backlog does not starve the healthy ones.
Third, a circuit breaker per partner counts failures and opens to fail fast, so stuck calls do not pile up. Fourth, transaction-ID idempotency ensures that retries after recovery return the stored result instead of debiting again. At National Bank of India, a cross-border payment had Rajesh's money debited into escrow and routed through two partners. When ICICI's API went down, the breaker opened and failed fast while HDFC kept working. The payment leg was retried idempotently until ICICI recovered, and the nightly reconciliation confirmed the ledger matched both partners.
The Saga holds money in a durable in-flight escrow so it is always accounted for. Each leg is a local step with a compensating reversal. If a leg ultimately cannot complete, the Saga refunds the sender from escrow. Reconciliation is the safety net that detects and corrects any divergence between the bank's ledger and each partner's ledger.
Remember one sentence: Guarantee no loss and no double-debit with per-partner bulkheads, circuit breakers, transaction-ID idempotency, and a Saga with compensations plus reconciliation, because external partners are REST endpoints and two-phase commit is both impossible and blocking.
---