Definition
- Distributed transaction (2PC) – a coordinator asks all participants to prepare, then tells them to commit only if all voted yes; preserves atomicity but is slow, holds locks, and the coordinator is a single point of failure.
- Kafka + XA – Kafka is not designed as an XA resource; forcing it into a global transaction is awkward and fragile.
- Atomicity – all participants commit together at a single instant, with no partial state ever visible.
- Recoverability – each system commits its own durable local transaction quickly and independently; overall consistency is reached afterward through eventual consistency, idempotent retries, and reconciliation.
- Decision rule – keep atomic invariants inside a single store; make everything crossing a system boundary eventually consistent.
Rajesh's transfer at National Bank of India must update the ledger and send an event. One idea is to make the ledger clerk and the Kafka courier shake hands and promise to both finish together or both cancel (2PC). But the clerk cannot close his ledger until the courier confirms, and if the courier is stuck in traffic the clerk is frozen. The smarter way: the clerk finishes his own ledger entry immediately, and a separate reliable process makes sure the event eventually goes out, with retries and a nightly reconciliation.
Understanding the Concepts
Why not use a distributed transaction across PostgreSQL and Kafka?
Two-phase commit (2PC) tries to extend ACID atomicity across multiple systems. A coordinator asks every participant "can you commit?" (prepare), and if all say yes, tells them "commit." This sounds clean but has severe practical problems.
What are the problems with 2PC?
- Kafka is not an XA resource -- forcing it into a global transaction is awkward and not how it is meant to be used.
- Slow -- multiple network round trips, and every participant holds locks from prepare to commit.
- Coordinator is a SPOF -- if it crashes after prepare but before commit, participants are stuck in-doubt holding locks.
- Availability coupling -- if any participant is down, the whole transaction cannot complete.
// 2PC: coordinator asks both participants to prepare
boolean dbPrepared = db.prepare(txn); // holds row locks
boolean kafkaPrepared = kafka.prepare(txn); // Kafka not designed for this!
if (dbPrepared && kafkaPrepared) {
db.commit(txn);
kafka.commit(txn);
// If coordinator crashes HERE → db committed, kafka in-doubt
}
What is the difference between atomicity and recoverability?
| Aspect | Atomicity (2PC) | Recoverability (Outbox) |
|---|---|---|
| Guarantee | All commit at one instant | Each commits locally; consistency reached eventually |
| Lock duration | Long (across network) | Short (local only) |
| Availability | Coupled (any participant down = all blocked) | Independent |
| Failure modes | Coordinator SPOF, in-doubt states | Duplicates (handled by idempotency) |
| Throughput | Low (network round trips + locks) | High (local commits + async relay) |
| Use case | Rare, short, must-be-atomic | Most cross-system workflows |
Which invariants truly need synchronous atomicity?
The debit and credit of a single transfer must both happen or neither -- but that lives entirely inside one database, so a single local transaction gives it for free.
The invariant that a transfer and its published event must both exist does NOT need to hold at a single instant. It only needs to hold eventually.
// This invariant is INSIDE one database → local ACID, no 2PC needed
@Transactional
public void transfer(String fromId, String toId, BigDecimal amount) {
Account from = accountRepo.findById(fromId).orElseThrow();
Account to = accountRepo.findById(toId).orElseThrow();
from.debit(amount);
to.credit(amount);
accountRepo.save(from);
accountRepo.save(to); // both in one local transaction
}
// This invariant CROSSES systems → outbox + eventual consistency
// DB commit + event published → outbox pattern, NOT 2PC
How does reconciliation work?
A periodic job compares the two systems and fixes drift.
// Nightly reconciliation
public void reconcile() {
for (Transfer t : transferRepo.findAllCommitted()) {
if (!eventStore.hasEventFor(t.getId())) {
outboxRepo.save(new OutboxEvent(t)); // re-publish missing event
log.warn("Reconciled missing event for transfer {}", t.getId());
}
}
}
What are common mistakes?
// WRONG: treating every cross-system relationship as if it needs atomicity
// → forces distributed transactions and all their pathologies
// CORRECT: keep atomic invariants inside one store
// → make everything crossing a boundary eventually consistent
What about idempotent retries?
// Safe retry: keyed by business ID
public void applyTransfer(String txnId, BigDecimal amount) {
if (ledger.alreadyApplied(txnId)) return; // idempotent
ledger.apply(txnId, amount);
}
What should a senior engineer know?
- The junior mistake is treating every cross-system relationship as if it needs instantaneous atomicity.
- The discipline is to keep atomic invariants inside a single store and make everything crossing a system boundary eventually consistent and reconcilable.
- Recoverability = durable local transactions + explicit eventual consistency + idempotent retries + reconciliation.
- Together these give effectively-once business behavior without an impossible distributed atomic commit.
Visual Overview
2PC (Distributed Transaction)
┌──────────────┐
│ Coordinator │
└──┬───────┬───┘
│prepare│prepare
▼ ▼
┌──────┐ ┌──────┐
│ DB │ │Kafka │ Both hold locks
└──────┘ └──────┘
▲ ▲
│commit │commit ← coordinator crash HERE = stuck
└───────┘
Recoverability (Outbox + Reconciliation)
┌──────────┐ local tx ┌──────┐
│ Service │ ──────────► │ DB │ committed instantly
└──────────┘ └──┬───┘
│ CDC relay (async)
┌────▼────┐
│ Kafka │ at-least-once
└─────────┘
│
┌────▼────────┐
│Reconciler │ nightly check
│ DB vs Kafka │
└─────────────┘
Key Takeaway
The dual-write problem tempts people to wrap the database write and the Kafka publish in a distributed transaction like two-phase commit, but that is the wrong tool. Kafka is not an XA resource, 2PC is slow with locks held across network round trips, the coordinator is a single point of failure, and it couples the availability of the database and the broker so if either is slow the other freezes.
The key insight is distinguishing atomicity from recoverability. Atomicity means all participants commit together at one instant. Recoverability means each system commits its own durable local transaction quickly and independently, and overall consistency is reached afterward through explicit eventual consistency, idempotent retries, and reconciliation. Most cross-system invariants only need recoverability.
The debit and credit of a single transfer must be atomic, but that lives inside one database so a local transaction gives it for free. The invariant that a transfer and its event must both exist does not need to hold at a single instant -- the outbox pattern, idempotent consumers, and a nightly reconciler give it eventually and safely.
Remember one sentence: never span two systems with a distributed transaction for the dual write; commit each system's own local transaction and reach consistency through eventual consistency, idempotent retries, and reconciliation, reserving true atomicity for invariants inside one store.
---