Definition
- No distributed ACID – Services own separate databases, so no single transaction can atomically reserve inventory, debit money, and confirm an order.
- Saga pattern – A sequence of local transactions, each committing in one service, with a compensating action for each step so failures undo earlier steps in reverse.
- Outbox pattern – Each service writes its event into an outbox table in the same local transaction, and a relay publishes it, making state and event atomic.
- Sync vs async – Create the order synchronously for an immediate ID; drive payment and inventory through Kafka events.
- Idempotency – A client-generated key ensures a retried create returns the existing order instead of a duplicate.
Imagine Rajesh buys something at National Bank of India's shopping platform. The bank must do three things in three separate departments: reserve the item in the warehouse, take the money from Rajesh's account, and confirm the order. Each department has its own ledger, so there is no single master pen that writes all three at once. Instead the bank follows a careful sequence: reserve, charge, confirm. If charging fails after the item was reserved, it must go back and release the reservation (compensation). And if Rajesh's phone retries because the network blinked, the bank recognizes the same order key and does not charge him twice.
Understanding the Concepts
Why can you not use a single ACID transaction across services?
Each service owns its own database. A two-phase commit across independent services is blocking, holds locks, and couples availability. The accepted pattern is the Saga: break the operation into local transactions, each committing in one service, and define a compensating action for each step.
package com.bank.order.model;
import java.math.BigDecimal;
import java.time.Instant;
public class Order {
private String orderId;
private String customerId; // e.g. "CUST-RAJESH-001"
private BigDecimal totalAmount;
private String status; // CREATED, INVENTORY_RESERVED, PAYMENT_CHARGED, CONFIRMED, FAILED
private String idempotencyKey;
private Instant createdAt;
public Order(String orderId, String customerId, BigDecimal totalAmount,
String idempotencyKey) {
this.orderId = orderId;
this.customerId = customerId;
this.totalAmount = totalAmount;
this.idempotencyKey = idempotencyKey;
this.status = "CREATED";
this.createdAt = Instant.now();
}
// Getters omitted for brevity
public String getOrderId() { return orderId; }
public String getStatus() { return status; }
public void setStatus(String status) { this.status = status; }
}
Where do transaction boundaries live?
Transaction boundaries live inside each service, never across them. The order service wraps its create in a local transaction. Payment wraps the debit. Inventory wraps the reservation. What spans services is the Saga logic and messaging.
package com.bank.order.service;
import com.bank.order.model.Order;
import java.math.BigDecimal;
public class OrderService {
private final OrderRepository orderRepository;
private final OutboxRepository outboxRepository;
public OrderService(OrderRepository orderRepository,
OutboxRepository outboxRepository) {
this.orderRepository = orderRepository;
this.outboxRepository = outboxRepository;
}
// Single local transaction: create order + write outbox event
public Order createOrder(String customerId, BigDecimal amount,
String idempotencyKey) {
// Idempotency check
Order existing = orderRepository.findByIdempotencyKey(idempotencyKey);
if (existing != null) {
return existing; // Return stored result, no duplicate
}
Order order = new Order(generateId(), customerId, amount, idempotencyKey);
orderRepository.save(order); // local DB write
outboxRepository.save("OrderCreated", order.getOrderId()); // same txn
return order;
}
private String generateId() {
return "ORD-" + System.currentTimeMillis();
}
}
How does the Saga handle payment and inventory failures?
The Saga steps are: reserve inventory, charge payment, confirm order. If any step fails, compensations run in reverse.
| Step | Service | Compensation |
|---|---|---|
| 1. Reserve inventory | Inventory | Release reservation |
| 2. Charge payment | Payment | Refund payment |
| 3. Confirm order | Order | Mark order FAILED |
package com.bank.order.saga;
import java.math.BigDecimal;
public class OrderSagaOrchestrator {
private final InventoryClient inventoryClient;
private final PaymentClient paymentClient;
private final OrderService orderService;
public OrderSagaOrchestrator(InventoryClient inventoryClient,
PaymentClient paymentClient,
OrderService orderService) {
this.inventoryClient = inventoryClient;
this.paymentClient = paymentClient;
this.orderService = orderService;
}
public void execute(String orderId, String customerId,
String productId, BigDecimal amount) {
// Step 1: Reserve inventory
boolean reserved = inventoryClient.reserve(orderId, productId);
if (!reserved) {
orderService.markFailed(orderId, "INVENTORY_UNAVAILABLE");
return;
}
// Step 2: Charge payment
boolean charged = paymentClient.charge(orderId, customerId, amount);
if (!charged) {
// Compensation: release inventory
inventoryClient.releaseReservation(orderId, productId);
orderService.markFailed(orderId, "PAYMENT_FAILED");
return;
}
// Step 3: Confirm
orderService.confirm(orderId);
}
}
How does synchronous vs asynchronous flow work?
Create the order synchronously so the customer gets an order ID immediately. Drive payment and inventory through Kafka as a Saga. Notify asynchronously.
package com.bank.order.controller;
import com.bank.order.model.Order;
import com.bank.order.service.OrderService;
import java.math.BigDecimal;
public class OrderController {
private final OrderService orderService;
private final KafkaPublisher kafkaPublisher;
public OrderController(OrderService orderService, KafkaPublisher kafkaPublisher) {
this.orderService = orderService;
this.kafkaPublisher = kafkaPublisher;
}
// Synchronous: customer gets order ID immediately
public Order placeOrder(String customerId, BigDecimal amount,
String idempotencyKey) {
Order order = orderService.createOrder(customerId, amount, idempotencyKey);
// Async: Saga starts via Kafka
kafkaPublisher.publish("order-events", order.getOrderId(), "OrderCreated");
return order; // Customer sees: "Order ORD-123 placed, processing..."
}
}
What is the outbox pattern and why is it needed?
The outbox pattern ensures the local state change and the event are atomic. Without it, you could commit the order but fail to publish the event, or publish but not commit.
package com.bank.order.outbox;
import java.time.Instant;
public class OutboxEvent {
private long id;
private String aggregateId; // orderId
private String eventType; // "OrderCreated"
private String payload; // JSON
private boolean published;
private Instant createdAt;
// A relay process polls unpublished events and sends to Kafka
// After successful publish, marks published = true
}
How does idempotent order creation work?
The client sends an idempotency key with every create request. The order service stores processed keys, so a retry returns the existing order instead of creating a new one.
package com.bank.order.service;
// WRONG: no idempotency check
public class OrderServiceWrong {
public Order createOrder(String customerId, java.math.BigDecimal amount) {
// Every call creates a new order, even retries!
return new Order(generateId(), customerId, amount, null);
}
}
// CORRECT: idempotency key prevents duplicates
// (See OrderService.createOrder above with idempotencyKey check)
How do you monitor the Saga?
Track each Saga instance's state and emit metrics for started, completed, compensated, and stuck Sagas. Alert on compensation rate and Sagas that have not progressed within expected time.
package com.bank.order.monitoring;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;
public class SagaMetrics {
private final AtomicLong started = new AtomicLong();
private final AtomicLong completed = new AtomicLong();
private final AtomicLong compensated = new AtomicLong();
private final AtomicLong stuck = new AtomicLong();
public void onSagaStarted() { started.incrementAndGet(); }
public void onSagaCompleted() { completed.incrementAndGet(); }
public void onSagaCompensated() { compensated.incrementAndGet(); }
public void onSagaStuck() { stuck.incrementAndGet(); }
public void printMetrics() {
System.out.printf("Sagas: started=%d, completed=%d, compensated=%d, stuck=%d%n",
started.get(), completed.get(), compensated.get(), stuck.get());
}
}
Sagas: started=1250, completed=1200, compensated=45, stuck=5
How do you scale the system?
Stateless services scale horizontally. Kafka partitions let consumers scale out, partitioned by order ID so all events for one order stay ordered. The money path stays strongly consistent while surrounding flows are eventual.
# Kafka partitioning by order ID: # Order ORD-001 → partition 0 → consumer-instance-A # Order ORD-002 → partition 1 → consumer-instance-B # All events for one order are ordered within its partition.
What are common interview follow-up questions?
How would you handle a payment that succeeds but the confirmation event is lost? What happens if the compensation itself fails? How do you test a Saga end-to-end? What is the difference between orchestration and choreography? When would you choose choreography over orchestration?
Visual Overview
┌──────────────────────────────────────────────────────────┐ │ ORDER SAGA FLOW (ORCHESTRATED) │ ├──────────────────────────────────────────────────────────┤ │ │ │ Client ──POST /orders──▶ Order Service (sync) │ │ │ │ │ ┌─────────▼──────────┐ │ │ │ Create Order + │ │ │ │ Outbox Event │ │ │ │ (one local txn) │ │ │ └─────────┬──────────┘ │ │ │ Kafka: OrderCreated │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Saga Orchestrator │ │ │ └───┬─────────────┬───┘ │ │ │ │ │ │ Step 1 │ Step 2 │ │ │ ┌─────────────▼┐ ┌────────▼────────┐ │ │ │ Inventory Svc │ │ Payment Svc │ │ │ │ Reserve stock │ │ Charge customer │ │ │ └──────┬────────┘ └──────┬──────────┘ │ │ │ │ │ │ Success? Success? │ │ YES │ NO │ │ │ │ ┌─────────────▼──────────┐ │ │ │ │ COMPENSATE: Release │ │ │ │ │ inventory reservation │ │ │ │ └────────────────────────┘ │ │ │ │ │ ┌─────▼──────────┐ │ │ │ Confirm Order │ │ │ │ (status=DONE) │ │ │ └────────────────┘ │ └──────────────────────────────────────────────────────────┘
Key Takeaway
An end-to-end payment and order system coordinates several services with separate databases, so the first thing Pankaj states in an interview is that no single ACID transaction can atomically reserve inventory, debit money, and confirm an order. The accepted pattern is the Saga: a sequence of local transactions, each committing in its own service's database, with a compensating action for each step. If a later step fails, the Saga undoes the earlier ones in reverse, and the outbox pattern ensures the state change and the event are always atomic.
At National Bank of India's shopping platform, when Rajesh places an order, the system creates the order synchronously so he immediately sees an order ID, then drives inventory reservation and payment through Kafka as an orchestrated Saga. If payment fails after inventory was reserved, the Saga releases the reservation so no stock is held for an unpaid order. If Priya's flaky mobile connection retries the order, the idempotency key ensures only one order is created and she is never double-charged.
The Saga orchestrator explicitly tells each service what to do next and triggers compensations on failure, which Pankaj prefers over choreography for the money path because explicit, auditable control matters. Every step and every compensation is idempotent because they will be retried after crashes. Monitoring tracks Saga instances with metrics for started, completed, compensated, and stuck, and a dead-letter queue captures poison events for investigation.
Remember one sentence: A payment and order system is an orchestrated Saga of local transactions with idempotent compensations, created synchronously for an immediate order ID and driven by Kafka for the rest, with the outbox pattern for atomic events and idempotency keys so retries never duplicate or double-charge.
---