Definition
- Strong consistency – every reader sees the latest write immediately; achievable within one service's ACID database.
- Eventual consistency – replicas and downstream services converge after a short delay; acceptable for statements, dashboards, search.
- 2PC/XA – atomic-commit protocol with prepare and commit phases; blocking, participants hold locks until coordinator decides.
- Saga – sequence of local transactions with compensating actions on failure; no global lock, eventual consistency.
- Transactional outbox – write business data and event row in one local transaction; relay publishes to Kafka at-least-once.
- Idempotent consumers – track processed event IDs so at-least-once delivery produces exactly-once effect.
Inside one department of National Bank of India, the ledger is kept perfectly balanced at every instant: that is strong consistency. But when Rajesh's transfer touches the accounts department, the fraud department, and the notification department, there is no single pen that writes all three books in one stroke. Instead each department does its own step and hands a note to the next. If a later step fails, earlier departments reverse their entry with a correcting note: that is a Saga. The notes are dropped reliably into a sealed mailbox so one can never be lost: that is the outbox.
Understanding the Concepts
What is strong vs eventual consistency?
Strong consistency guarantees that once a write commits, every subsequent read sees it. Eventual consistency guarantees only that all copies converge after some delay.
package com.bank.consistency;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
import java.util.logging.Logger;
@Service
public class AccountService {
private static final Logger LOG = Logger.getLogger(AccountService.class.getName());
private final AccountRepository repository;
public AccountService(AccountRepository repository) {
this.repository = repository;
}
@Transactional // Strong consistency: ACID within one database
public void debit(String accountId, BigDecimal amount) {
var account = repository.findById(accountId).orElseThrow();
account.debit(amount); // balance never goes negative
repository.save(account);
LOG.info("Strong consistency: debit committed for " + accountId);
}
}
Strong consistency: within one service's database.
Rajesh's balance is accurate the instant the transaction commits.
Eventual consistency: across services.
Priya's statement service gets the update seconds later via events.
Acceptable for derived data (statements, dashboards, search).
Why is 2PC problematic for microservices?
Two-phase commit provides atomicity but is blocking. Participants hold locks from prepare until the final decision. If the coordinator crashes, participants are stuck.
2PC protocol:
Phase 1 (Prepare): Coordinator asks all participants to prepare.
Each participant locks rows and votes YES or NO.
Phase 2 (Commit): If all voted YES -> commit. Otherwise -> abort.
Danger: Coordinator crashes after unanimous YES but before broadcasting COMMIT.
All participants stuck with locks held indefinitely.
High-throughput bank: locked rows block all other transfers.
Why microservices avoid 2PC:
- Blocking: locks held across network
- Availability: coupled to slowest participant
- Most services (REST, Kafka, NoSQL) are not XA-capable
How does the Saga pattern work?
A Saga models the operation as a sequence of local transactions with compensating actions on failure.
package com.bank.saga;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.logging.Logger;
@Service
public class TransferSagaOrchestrator {
private static final Logger LOG = Logger.getLogger(TransferSagaOrchestrator.class.getName());
private final AccountClient accountClient;
private final FraudClient fraudClient;
public TransferSagaOrchestrator(AccountClient accountClient, FraudClient fraudClient) {
this.accountClient = accountClient;
this.fraudClient = fraudClient;
}
public void execute(String fromId, String toId, BigDecimal amount) {
try {
// Step 1: Debit sender (local tx in Account Service)
accountClient.debit(fromId, amount);
LOG.info("Step 1: Debited " + fromId);
// Step 2: Fraud check
fraudClient.check(fromId, toId, amount);
LOG.info("Step 2: Fraud check passed");
// Step 3: Credit receiver (local tx in Account Service)
accountClient.credit(toId, amount);
LOG.info("Step 3: Credited " + toId);
} catch (Exception e) {
// Compensation: re-credit sender
accountClient.credit(fromId, amount);
LOG.warning("Saga compensated: re-credited " + fromId
+ " reason: " + e.getMessage());
}
}
}
Saga execution (success):
Step 1: Debit SAV2001 -> committed locally
Step 2: Fraud check -> passed
Step 3: Credit SAV2002 -> committed locally
Saga execution (failure at step 3):
Step 1: Debit SAV2001 -> committed locally
Step 2: Fraud check -> passed
Step 3: Credit SAV2002 -> FAILED
Compensation: Credit SAV2001 -> money returned to Rajesh
What is the transactional outbox pattern?
The outbox writes business data and an event row in one local transaction. A relay publishes the outbox rows to Kafka.
package com.bank.outbox;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
import java.util.logging.Logger;
@Service
public class OutboxTransferService {
private static final Logger LOG = Logger.getLogger(OutboxTransferService.class.getName());
private final AccountRepository accountRepo;
private final OutboxRepository outboxRepo;
public OutboxTransferService(AccountRepository accountRepo, OutboxRepository outboxRepo) {
this.accountRepo = accountRepo;
this.outboxRepo = outboxRepo;
}
@Transactional // Both writes in ONE local transaction
public void debit(String accountId, BigDecimal amount) {
var account = accountRepo.findById(accountId).orElseThrow();
account.debit(amount);
accountRepo.save(account);
// Write event to outbox in SAME transaction
outboxRepo.save(new OutboxEvent(
"AccountDebited", accountId, amount.toString()));
LOG.info("Debit + outbox event committed atomically for " + accountId);
}
// Relay (polling or Debezium CDC) publishes outbox rows to Kafka.
// At-least-once delivery -> consumers must be idempotent.
}
Dual-write problem:
1. Commit to DB
2. Publish to Kafka <- crash here = DB updated but no event published
Outbox solution:
1. Commit to DB + outbox table (one transaction)
2. Relay publishes outbox rows to Kafka (separate process)
Both commit or roll back together. Never inconsistent.
How do consumers handle at-least-once delivery?
Consumers track processed event IDs and ignore duplicates.
package com.bank.consumer;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import java.util.logging.Logger;
@Component
public class IdempotentNotificationConsumer {
private static final Logger LOG = Logger.getLogger(IdempotentNotificationConsumer.class.getName());
private final ProcessedEventRepository processedEventRepo;
public IdempotentNotificationConsumer(ProcessedEventRepository processedEventRepo) {
this.processedEventRepo = processedEventRepo;
}
@KafkaListener(topics = "payment-events", groupId = "notification-group")
public void onEvent(PaymentEvent event) {
if (processedEventRepo.existsById(event.getEventId())) {
LOG.info("Duplicate event ignored: " + event.getEventId());
return; // idempotent: skip duplicate
}
// Process the event (send SMS)
sendSms(event);
processedEventRepo.save(new ProcessedEvent(event.getEventId()));
LOG.info("Event processed: " + event.getEventId());
}
private void sendSms(PaymentEvent event) { /* ... */ }
}
At-least-once delivery -> possible duplicate events.
Idempotent consumer -> checks processedEventRepo -> ignores duplicates.
Result: exactly-once effect from at-least-once delivery.
How do these approaches compare? (comparison table)
| Approach | Consistency | Blocking | Availability | Use Case |
|---|---|---|---|---|
| Local ACID | Strong | No (single DB) | High | Within one service |
| 2PC/XA | Strong (cross-resource) | Yes (locks held) | Low | Rare, XA-capable resources |
| Saga + Outbox | Eventual | No | High | Cross-service operations |
| Event-driven | Eventual | No | High | Derived data, notifications |
What should a senior engineer know?
Choose consistency per data type, not globally. The authoritative money stays strongly consistent inside the owning service. Everything cross-service is a Saga driven by outbox events with idempotent consumers. Reserve 2PC only for XA-capable resources where atomicity is non-negotiable.
What are common mistakes?
The most common mistake is attempting strong consistency across services with synchronous two-phase calls, which couples services and kills availability. Another is forgetting idempotency on the consumer, which makes at-least-once delivery cause double credits.
package com.bank.consistency;
public class ConsistencyMistakes {
public static void main(String[] args) {
// WRONG: synchronous distributed transaction across 3 services
// accountService.debit() -> fraudService.check() -> notificationService.send()
// All in one HTTP call chain. One slow service blocks everything.
System.out.println("WRONG: sync distributed TX across services");
System.out.println(" Couples availability to slowest service");
// CORRECT: saga with local transactions and outbox events
// Each service commits locally, publishes via outbox, compensates on failure
System.out.println("CORRECT: saga + outbox + idempotent consumers");
// WRONG: non-idempotent consumer
// Event replayed -> customer credited twice!
System.out.println("WRONG: consumer without idempotency check");
System.out.println(" Event replayed -> double credit!");
// CORRECT: check processedEventRepo before processing
System.out.println("CORRECT: check event ID before processing");
}
}
Expected output:
WRONG: sync distributed TX across services
Couples availability to slowest service
CORRECT: saga + outbox + idempotent consumers
WRONG: consumer without idempotency check
Event replayed -> double credit!
CORRECT: check event ID before processing
Visual Overview
Consistency Approaches: WITHIN ONE SERVICE ACROSS SERVICES ┌───────────────────┐ ┌──────────────────────────┐ │ Local ACID │ │ Saga + Outbox + Events │ │ @Transactional │ │ │ │ Strong consistency │ │ Step 1: Debit (local tx) │ │ │ │ Step 2: Outbox event │ │ Balance is correct │ │ Step 3: Relay -> Kafka │ │ the instant commit │ │ Step 4: Credit (local tx)│ │ happens. │ │ Compensation if failure │ │ │ │ │ │ │ │ Eventual consistency │ │ │ │ Idempotent consumers │ └───────────────────┘ └──────────────────────────┘
Key Takeaway
Data consistency in microservices begins with accepting that a single ACID transaction can no longer span multiple services, so you choose a consistency model deliberately. Strong consistency is achievable inside one service's database and is what you use for the authoritative money. Eventual consistency is what you get across services, and it is fine for derived data like statements and search indexes.
For cross-service operations, the Saga pattern models the operation as local transactions with compensating actions. At National Bank of India, a transfer from Rajesh to Priya runs as a saga: debit, fraud check, credit, and if credit fails, compensate by re-crediting Rajesh. The transactional outbox ensures events are published reliably by writing both the business data and the event in one local transaction, with a relay publishing to Kafka. Consumers are idempotent, tracking event IDs so at-least-once delivery produces exactly-once effect.
Two-phase commit provides cross-resource atomicity but is avoided for cross-service operations because it is blocking, couples availability to the slowest participant, and most services are not XA-capable.
Remember one sentence: keep money strongly consistent inside the owning service, coordinate multi-service operations with Sagas and the transactional outbox, make consumers idempotent, and accept eventual consistency for everything derived.
---