Definition
- Distributed transaction -- a transaction that spans more than one resource (two databases, or a database and a message broker) and must commit or roll back all of them atomically.
- JTA -- Java Transaction API, the standard for coordinating transactions across multiple resources.
- XA -- the protocol that lets an external transaction manager coordinate a resource's commit; XA-compliant resources support
prepare()andcommit()calls. - Two-phase commit (2PC) -- the protocol where a coordinator first asks all resources to prepare and vote, then tells them all to commit if all voted yes or roll back if any voted no.
- Saga -- a pattern that breaks a distributed transaction into local transactions with compensating actions; comes in orchestration (central coordinator) and choreography (event-driven) styles.
- Transactional Outbox -- write the business row and an outbox event in the same local transaction, then relay the event to Kafka asynchronously.
Imagine the National Bank of India must move money from Rajesh's savings account and simultaneously update a separate insurance system. Two-phase commit is like a wedding officiant asking both the bride and groom "do you agree?" and only finalizing the marriage if both say yes. The problem is everyone must stand frozen at the altar holding their breath (holding locks) while the officiant waits for slow replies. If the officiant faints mid-ceremony (coordinator crash), everyone is stuck. In microservices we instead do the steps one at a time and, if a later step fails, run a "sorry, undo that" step: that is Saga.
Understanding the Concepts
What are distributed transactions?
A distributed transaction spans more than one resource. A resource can be a database, a message broker, or any system that supports transactional operations. The goal is to commit or roll back all resources atomically: either every resource commits, or every resource rolls back. No resource should commit while another rolls back.
In a banking system, a transfer from Rajesh's savings account (in a PostgreSQL database) to an insurance premium payment (in an Oracle database) is a distributed transaction. Both databases must agree on the outcome.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class InsurancePremiumService {
private final SavingsRepository savingsRepo; // PostgreSQL
private final InsuranceRepository insuranceRepo; // Oracle
public InsurancePremiumService(SavingsRepository savingsRepo,
InsuranceRepository insuranceRepo) {
this.savingsRepo = savingsRepo;
this.insuranceRepo = insuranceRepo;
}
// This requires a distributed transaction to be atomic
@Transactional // But a single @Transactional manages ONE DB only!
public void payPremium(String accountId, String policyId,
BigDecimal amount) {
savingsRepo.debit(accountId, amount); // PostgreSQL TX
insuranceRepo.recordPayment(policyId, amount); // Oracle -- NOT in same TX!
}
}
How does two-phase commit work?
Two-phase commit (2PC) is the protocol that makes distributed transactions atomic. It requires a coordinator (transaction manager) and two or more participants (XA resources).
In phase one (prepare), the coordinator asks each resource: "Can you commit?" Each resource does all the work, writes it to a durable log, locks the rows, and votes YES or NO. It does not commit yet.
In phase two (commit/rollback), if every resource voted YES, the coordinator tells them all to commit. If any resource voted NO, the coordinator tells them all to roll back.
package com.bank.config;
import com.atomikos.jdbc.AtomikosDataSourceBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.transaction.jta.JtaTransactionManager;
@Configuration
public class DistributedTxConfig {
@Bean
public AtomikosDataSourceBean savingsXaDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
ds.setUniqueResourceName("savings");
ds.setXaDataSourceClassName(
"org.postgresql.xa.PGXADataSource");
ds.getXaProperties().setProperty("url",
"jdbc:postgresql://db1:5432/savings_db");
return ds;
}
@Bean
public AtomikosDataSourceBean insuranceXaDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
ds.setUniqueResourceName("insurance");
ds.setXaDataSourceClassName(
"oracle.jdbc.xa.client.OracleXADataSource");
ds.getXaProperties().setProperty("URL",
"jdbc:oracle:thin:@db2:1521:INS");
return ds;
}
@Bean
public JtaTransactionManager transactionManager() {
return new JtaTransactionManager(); // Atomikos auto-detected
}
}
Two-phase commit flow:
Coordinator (Atomikos) PostgreSQL Oracle
│ │ │
│──── PREPARE ──────────────▶│ │
│ │── vote YES ──▶│
│──── PREPARE ──────────────────────────────▶│
│ │── vote YES
│ │
│ All voted YES │
│──── COMMIT ───────────────▶│ │
│──── COMMIT ───────────────────────────────▶│
│ │ │
│ Both committed atomically │ │
Why is two-phase commit discouraged in microservices?
Two-phase commit has four problems in a microservices architecture.
First, it blocks. Between prepare and commit, every resource holds row-level locks and keeps its transaction open across network round-trips. For a bank processing thousands of transfers per second, holding row locks across a network call to another service destroys throughput.
Second, it couples services. Every participant must be available at commit time. If one service is down, the entire transaction cannot complete, violating the independence that microservices are built for.
Third, the coordinator is a single point of failure. If it crashes after prepare but before commit, resources are left in an "in-doubt" state, holding locks until the coordinator recovers.
Fourth, many cloud data stores and message brokers (like Kafka) do not support XA at all.
// Problem illustration: blocking between phases
// Timeline during 2PC:
// 0 ms -- coordinator sends PREPARE to both DBs
// 50 ms -- PostgreSQL votes YES (holds row lock on SAV2001)
// 200 ms -- Oracle votes YES (holds row lock on POLICY001)
// 200 ms -- coordinator sends COMMIT
// 250 ms -- PostgreSQL commits, releases lock
// 400 ms -- Oracle commits, releases lock
//
// For 400 ms, both rows are LOCKED
// No other transaction can touch SAV2001 or POLICY001
What is the Saga pattern?
A Saga breaks one distributed transaction into a sequence of local transactions, each in its own service, each committing independently. If a later step fails, the Saga runs compensating transactions to undo the earlier steps. There is no global lock and no coordinator holding everything open.
package com.bank.saga;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class PremiumPaymentSaga {
private final SavingsService savingsService;
private final InsuranceService insuranceService;
public PremiumPaymentSaga(SavingsService savingsService,
InsuranceService insuranceService) {
this.savingsService = savingsService;
this.insuranceService = insuranceService;
}
// Saga orchestrator: coordinates local transactions
public void payPremium(String accountId, String policyId,
BigDecimal amount) {
// Step 1: debit savings (local TX)
savingsService.debit(accountId, amount);
try {
// Step 2: record insurance payment (local TX)
insuranceService.recordPayment(policyId, amount);
} catch (Exception e) {
// Compensation: credit savings back
savingsService.credit(accountId, amount);
throw new SagaFailedException(
"Insurance payment failed, savings refunded", e);
}
}
}
Saga success:
Step 1: debit SAV2001 Rs.5000 -- committed
Step 2: record POLICY001 Rs.5000 -- committed
Result: both committed independently
Saga failure:
Step 1: debit SAV2001 Rs.5000 -- committed
Step 2: record POLICY001 fails
Compensation: credit SAV2001 Rs.5000 -- committed
Result: money returned to Rajesh
What is the difference between Orchestration and Choreography Sagas?
| Aspect | Orchestration | Choreography |
|---|---|---|
| Central brain | Yes: a Saga coordinator | No: services react to events |
| Coupling | Services coupled to coordinator | Services coupled to events |
| Traceability | Easy: coordinator has full state | Hard: events scattered across services |
| Complexity | Coordinator is complex | Each service has event handlers |
| Failure handling | Coordinator triggers compensations | Each service publishes failure events |
| Best for | Complex flows with many steps | Simple flows with 2-3 steps |
// Orchestration: coordinator calls each service in order
public void payPremiumOrchestrated(String accountId,
String policyId, BigDecimal amount) {
savingsService.debit(accountId, amount); // step 1
insuranceService.recordPayment(policyId, amount); // step 2
// Coordinator knows the full state
}
// Choreography: services react to events
// SavingsService publishes DebitCompletedEvent
// InsuranceService listens for DebitCompletedEvent
// No central coordinator
What is the Transactional Outbox pattern?
The Outbox pattern solves a subtle problem: you cannot atomically write to your database and publish to Kafka in one transaction because Kafka does not support XA. Instead, in the same local database transaction, you write your business row and insert a row into an outbox table. A separate relay process reads unsent outbox rows and publishes them to Kafka.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class OutboxPremiumService {
private final SavingsRepository savingsRepo;
private final OutboxRepository outboxRepo;
public OutboxPremiumService(SavingsRepository savingsRepo,
OutboxRepository outboxRepo) {
this.savingsRepo = savingsRepo;
this.outboxRepo = outboxRepo;
}
@Transactional // single local TX
public void payPremium(String accountId, String policyId,
BigDecimal amount) {
savingsRepo.debit(accountId, amount);
outboxRepo.save(new OutboxEvent(
"INSURANCE_PAYMENT",
"{\"policyId\":\"" + policyId
+ "\",\"amount\":" + amount + "}"));
// Both committed atomically
}
}
Outbox flow:
1. debit savings + insert outbox row -- ONE local TX
2. Relay process (Debezium CDC or scheduler) reads outbox rows
3. Relay publishes to Kafka topic "insurance-payments"
4. Insurance service consumes event, records payment
5. Relay marks outbox row as sent
Guarantee: at-least-once delivery (event published
if and only if business data committed)
What are common mistakes?
The first mistake is reaching for JTA/XA in a microservices architecture. Two-phase commit creates tight coupling, blocks under load, and many cloud-native databases do not support XA.
// WRONG: using XA across microservices
// Each microservice has its own database
// XA locks span network boundaries
// Coordinator crash leaves both DBs locked
// CORRECT: use Saga + Outbox
// Each service commits locally
// Events relay asynchronously
// Compensations handle failures
The second mistake is not making Saga steps idempotent. In an eventually consistent system, messages can be delivered more than once. If the insurance service receives the same payment event twice, it must not charge Rajesh twice.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class InsurancePaymentService {
private final InsuranceRepository insuranceRepo;
public InsurancePaymentService(InsuranceRepository insuranceRepo) {
this.insuranceRepo = insuranceRepo;
}
// WRONG: not idempotent -- duplicate event charges twice
@Transactional
public void recordPaymentWrong(String policyId, BigDecimal amount) {
insuranceRepo.addPayment(policyId, amount);
}
// CORRECT: idempotent -- uses idempotency key
@Transactional
public void recordPaymentCorrect(String eventId, String policyId,
BigDecimal amount) {
if (insuranceRepo.existsByEventId(eventId)) {
return; // already processed
}
insuranceRepo.addPayment(policyId, amount, eventId);
}
}
What should a senior engineer know?
A senior engineer knows the CAP theorem tradeoff: two-phase commit prioritizes Consistency over Availability, while Saga prioritizes Availability over strict Consistency. In a banking system, brief inconsistency (eventually consistent) is acceptable for most operations, but some operations (like regulatory reporting) may require strong consistency.
Senior engineers also know about the Saga execution coordinator (SEC) pattern, where the coordinator persists its state in a durable store so it can recover from crashes and resume from the last completed step. They know about dead letter queues for events that fail after all retries.
package com.bank.saga;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class SagaCoordinator {
private final SagaStateRepository sagaRepo;
public SagaCoordinator(SagaStateRepository sagaRepo) {
this.sagaRepo = sagaRepo;
}
@Transactional
public void executeStep(String sagaId, int stepIndex) {
SagaState state = sagaRepo.findById(sagaId).orElseThrow();
state.setCurrentStep(stepIndex);
state.setStatus("IN_PROGRESS");
sagaRepo.save(state);
// If coordinator crashes here, restart reads state
// and resumes from currentStep
}
}
What are common interview follow-up questions?
- What happens if the coordinator crashes between phase one and phase two?
- How does the Outbox pattern guarantee at-least-once delivery?
- When would you still choose two-phase commit over Saga?
- How do you handle a compensation that itself fails in a Saga?
- What is the difference between Saga and CQRS?
Visual Overview
Two-Phase Commit (2PC):
┌──────────────┐ ┌──────────┐ ┌──────────┐
│ Coordinator │ │ DB 1 │ │ DB 2 │
│ (Atomikos) │ │ (Postgres)│ │ (Oracle) │
└──────┬───────┘ └─────┬────┘ └─────┬────┘
│ Phase 1: PREPARE │ │
│──────────────────▶│ │
│ vote YES ◀──────│ │
│──────────────────────────────────▶ │
│ vote YES ◀────────│
│ │
│ Phase 2: COMMIT │
│──────────────────▶│ │
│──────────────────────────────────▶ │
│ Both committed │ │
Saga Pattern:
┌──────────────┐ ┌──────────┐ ┌──────────┐
│ Orchestrator │ │ Savings │ │ Insurance│
└──────┬───────┘ └─────┬────┘ └─────┬────┘
│ Step 1: debit │ │
│──────────────────▶│ │
│ committed ◀──────│ │
│ Step 2: payment │ │
│──────────────────────────────────▶ │
│ committed ◀────────────────────────│
│ │
│ If step 2 fails: │
│ Compensate: credit savings back │
│──────────────────▶│ │
Key Takeaway
A distributed transaction spans more than one resource and must commit or roll back all of them together. The classic mechanism is JTA coordinating XA resources with two-phase commit. In phase one, the coordinator asks every resource to prepare and vote. In phase two, it tells them all to commit if everyone voted yes, or roll back if anyone voted no. That gives strict atomicity, but between the two phases every resource holds its locks and keeps its transaction open across network calls.
That cost is why two-phase commit is discouraged in microservices. Holding row locks across a network round-trip destroys throughput under load, every participant must be available at commit time which breaks service independence, and the coordinator is a single point of failure. If it crashes between phases, resources are left in doubt holding locks until it recovers. Many modern stores and Kafka do not even support XA.
In the National Bank of India premium payment system, Pankaj moved from JTA to a Saga. He broke the payment into local transactions: debit Rajesh's savings with a local commit, then record the insurance payment with a local commit. If the insurance step failed, a compensating transaction credited Rajesh back. Combined with the Outbox pattern, the business row and an outbox event committed atomically in one local transaction, and a relay published the event to Kafka. The tradeoff was eventual consistency, so every operation and compensation was made idempotent.
Remember one sentence: two-phase commit buys strict atomicity at the price of availability and scale, so microservices prefer Saga plus Outbox with eventual consistency.
---