Definition
- Dual-write problem -- writing to the DB and publishing to a broker as two non-atomic operations; a crash between them leaves the world inconsistent.
- Transactional Outbox -- write the domain change and an event row into an outbox table in the same local ACID transaction; both commit or roll back together.
- Relay -- a separate process publishes outbox rows to Kafka; implemented via polling or Change Data Capture (Debezium tailing the write-ahead log).
- At-least-once delivery -- the relay may republish on restart; consumers must be idempotent to absorb duplicates.
- Idempotent consumers -- record processed event IDs; ignore duplicates; turns at-least-once into exactly-once effect.
- Saga enabler -- the outbox makes each Saga step reliable by capturing its event atomically with its state change.
When Rajesh transfers money at National Bank of India, two things must both be true: his balance is updated in the ledger, and a notice is posted so the SMS and statement services learn about it. If the clerk updates the ledger and then, before posting the notice, the power goes out, the money moved but nobody was told. The outbox fixes this: the clerk writes the ledger entry and drops the notice into a sealed outbox drawer in the exact same locked step. A separate courier later empties the outbox and delivers every notice. If the courier delivers one twice, the recipients are smart enough to ignore the duplicate.
Understanding the Concepts
What is the dual-write problem?
The dual-write problem appears whenever a service must write to its database and publish a message to a broker like Kafka as two separate operations. These are two independent writes with no shared transaction, so there are failure interleavings that leave the world inconsistent.
package com.bank.transfer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import java.math.BigDecimal;
public class DualWriteProblem {
private final JdbcTemplate db;
private final KafkaTemplate<String, String> kafka;
public DualWriteProblem(JdbcTemplate db, KafkaTemplate<String, String> kafka) {
this.db = db;
this.kafka = kafka;
}
// WRONG: two separate writes, not atomic
public void transfer(String fromId, String toId, BigDecimal amount) {
db.update("UPDATE accounts SET balance = balance - ? WHERE id = ?",
amount, fromId);
db.update("UPDATE accounts SET balance = balance + ? WHERE id = ?",
amount, toId);
// --- CRASH WINDOW ---
// If the process crashes HERE, the money moved
// but no event was ever published.
kafka.send("transfer.completed", fromId,
fromId + ":" + toId + ":" + amount);
}
}
Scenario:
1. DB commit succeeds: Rajesh debited, Priya credited
2. Process crashes before kafka.send()
3. Event never published --> SMS service never learns
4. Priya's statement is stale, no notification sent
Why can you not fix it by reordering?
Publishing to Kafka first and then committing to the DB does not fix the problem; it moves the window. If Kafka receives the event but the DB commit fails, you have announced a transfer that never happened and consumers act on a phantom.
package com.bank.transfer;
public class ReorderingDoesNotHelp {
public static void main(String[] args) {
System.out.println("Order 1: DB then Kafka --> crash before Kafka = lost event");
System.out.println("Order 2: Kafka then DB --> crash before DB = phantom event");
System.out.println("Neither order closes the window.");
System.out.println("You need ATOMIC capture of both in one step.");
}
}
Expected output:
Order 1: DB then Kafka --> crash before Kafka = lost event
Order 2: Kafka then DB --> crash before DB = phantom event
Neither order closes the window.
You need ATOMIC capture of both in one step.
How does the Transactional Outbox solve it?
Add an outbox table in the service's own database. In the single local transaction that performs the business change, also insert a row into the outbox table describing the event. Because both writes are in the same local ACID transaction, they commit together or roll back together.
package com.bank.transfer;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
import java.util.UUID;
public class TransactionalOutbox {
private final JdbcTemplate db;
public TransactionalOutbox(JdbcTemplate db) {
this.db = db;
}
@Transactional
public void transfer(String fromId, String toId, BigDecimal amount) {
// Business change
db.update("UPDATE accounts SET balance = balance - ? WHERE id = ?",
amount, fromId);
db.update("UPDATE accounts SET balance = balance + ? WHERE id = ?",
amount, toId);
// Outbox row in the SAME transaction
String eventId = UUID.randomUUID().toString();
String payload = fromId + ":" + toId + ":" + amount;
db.update(
"INSERT INTO outbox (event_id, aggregate_key, topic, payload) " +
"VALUES (?, ?, ?, ?)",
eventId, fromId, "transfer.completed", payload);
// Both commit atomically or both roll back
}
}
Expected behavior:
Single local transaction: debit + credit + outbox row
All commit together --> no lost event, no phantom event
Process crashes after commit --> event safely in outbox table
What does the outbox table look like?
The outbox table stores events waiting to be published. Each row has a unique event ID for deduplication, an aggregate key for Kafka partitioning, and a published flag for the relay.
CREATE TABLE outbox (
event_id VARCHAR(36) PRIMARY KEY,
aggregate_key VARCHAR(50) NOT NULL,
topic VARCHAR(100) NOT NULL,
payload TEXT NOT NULL,
published BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
package com.bank.transfer;
public class OutboxSchema {
public static void main(String[] args) {
System.out.println("event_id: UUID for deduplication");
System.out.println("aggregate_key: Kafka partition key (account ID)");
System.out.println("topic: Kafka topic name");
System.out.println("payload: serialized event data");
System.out.println("published: false until relay publishes it");
}
}
Expected output:
event_id: UUID for deduplication
aggregate_key: Kafka partition key (account ID)
topic: Kafka topic name
payload: serialized event data
published: false until relay publishes it
How does the relay publish outbox rows?
Two common implementations: polling and Change Data Capture.
Polling: a background job periodically selects unpublished rows, publishes each to Kafka, and marks it published. Simple and portable but adds polling latency and database load.
CDC (Debezium): tails the database's write-ahead log and streams every committed outbox insert to Kafka with very low latency and no polling load. Preferred at scale.
package com.bank.transfer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import java.util.List;
import java.util.Map;
public class PollingRelay {
private final JdbcTemplate db;
private final KafkaTemplate<String, String> kafka;
public PollingRelay(JdbcTemplate db, KafkaTemplate<String, String> kafka) {
this.db = db;
this.kafka = kafka;
}
@Scheduled(fixedDelay = 500) // every 500ms
public void publishOutbox() {
List<Map<String, Object>> rows = db.queryForList(
"SELECT event_id, aggregate_key, topic, payload " +
"FROM outbox WHERE published = FALSE ORDER BY created_at LIMIT 100");
for (Map<String, Object> row : rows) {
kafka.send((String) row.get("topic"),
(String) row.get("aggregate_key"),
(String) row.get("payload"));
db.update("UPDATE outbox SET published = TRUE WHERE event_id = ?",
row.get("event_id"));
}
}
}
Why must consumers be idempotent?
The relay delivers at-least-once, not exactly-once. It might publish a row to Kafka and crash before marking it published. On restart, it republishes the same event. Consumers must record processed event IDs and ignore duplicates.
package com.bank.notification;
import org.springframework.kafka.annotation.KafkaListener;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
public class IdempotentConsumer {
// In production, use a database table instead of in-memory set
private final Set<String> processedIds = ConcurrentHashMap.newKeySet();
@KafkaListener(topics = "transfer.completed")
public void onTransferCompleted(String eventId, String payload) {
if (processedIds.contains(eventId)) {
System.out.println("Duplicate event " + eventId + " -- ignored");
return;
}
processedIds.add(eventId);
System.out.println("Processing event " + eventId + ": " + payload);
// Send SMS to Priya
}
}
Expected output:
Processing event abc-123: SAV2001:SAV2002:5000.00
Duplicate event abc-123 -- ignored
How is polling different from CDC (Debezium)?
| Aspect | Polling | CDC (Debezium) |
|---|---|---|
| Latency | Polling interval (100ms-1s) | Near real-time (milliseconds) |
| DB load | SELECT every interval | Reads existing WAL, near zero overhead |
| Ordering | Must ORDER BY created_at | Preserves commit order from WAL |
| Complexity | Simple (just SQL + scheduler) | Requires Debezium + connector setup |
| Scale | Adds load as event rate grows | Scales with WAL throughput |
package com.bank.transfer;
public class PollingVsCdc {
public static void main(String[] args) {
System.out.println("Polling: simple, portable, adds DB load");
System.out.println("CDC: low latency, no polling load, needs Debezium");
System.out.println("Use polling for low volume, CDC for production scale");
}
}
Expected output:
Polling: simple, portable, adds DB load
CDC: low latency, no polling load, needs Debezium
Use polling for low volume, CDC for production scale
What are common mistakes?
The most common mistake is publishing to Kafka outside the transaction, re-introducing the dual-write window.
package com.bank.transfer;
public class CommonMistakes {
public static void main(String[] args) {
// WRONG: publishing to Kafka after the transaction commits
System.out.println("WRONG: @Transactional { db.update(); } kafka.send();");
System.out.println("Crash between commit and send = lost event");
// CORRECT: outbox row inside the transaction
System.out.println("CORRECT: @Transactional { db.update(); outbox.insert(); }");
System.out.println("Relay publishes later; event is never lost");
}
}
Expected output:
WRONG: @Transactional { db.update(); } kafka.send();
Crash between commit and send = lost event
CORRECT: @Transactional { db.update(); outbox.insert(); }
Relay publishes later; event is never lost
What should a senior engineer know?
- Keep the outbox lean: prune or archive published rows so the table does not grow unbounded.
- Include a stable event ID and an aggregate key so consumers can dedupe and Kafka can partition by aggregate for per-aggregate ordering.
- Prefer CDC with Debezium at scale for latency and to avoid loading the primary database with polling.
- The outbox is what makes a Saga reliable: every Saga step's local transaction writes its outbox event atomically.
package com.bank.transfer;
public class SeniorInsights {
public static void main(String[] args) {
System.out.println("1. Prune published outbox rows (retention policy)");
System.out.println("2. Include stable event_id for consumer deduplication");
System.out.println("3. Use aggregate_key as Kafka partition key");
System.out.println("4. Prefer Debezium CDC at production scale");
System.out.println("5. Outbox + idempotent consumers = reliable event publishing");
}
}
Expected output:
1. Prune published outbox rows (retention policy)
2. Include stable event_id for consumer deduplication
3. Use aggregate_key as Kafka partition key
4. Prefer Debezium CDC at production scale
5. Outbox + idempotent consumers = reliable event publishing
Visual Overview
Dual-Write Problem
┌──────────┐ ┌──────────┐
│ Transfer │ │ │
│ Service │────►│ DB │ Step 1: commit
│ │ └──────────┘
│ │ --- CRASH WINDOW ---
│ │ ┌──────────┐
│ │────►│ Kafka │ Step 2: publish (never happens)
└──────────┘ └──────────┘
Money moved but event lost!
Transactional Outbox (fix)
┌──────────┐ ┌─────────────────────┐
│ Transfer │ │ DB │
│ Service │────►│ accounts + outbox │ ONE transaction
└──────────┘ └─────────┬───────────┘
│
┌─────────▼───────────┐
│ Relay (poll or CDC) │
└─────────┬───────────┘
│
┌─────────▼───────────┐
│ Kafka │
│ at-least-once │
└─────────────────────┘
│
┌─────────▼───────────┐
│ Idempotent Consumer │
│ dedupe by event_id │
└─────────────────────┘
Key Takeaway
The dual-write problem arises when a service must update its database and publish a message to Kafka as two separate operations. A crash between the two leaves the world inconsistent: the money moved but no event was published, or an event was published but the money did not move. Reordering the two steps only moves the failure window; it never closes it.
The Transactional Outbox closes the window by turning two writes into one. The business change and an outbox row are written in the same local ACID transaction, so they commit together or roll back together. A separate relay then publishes outbox rows to Kafka, either by polling or by Debezium tailing the write-ahead log. The relay delivers at-least-once, so consumers must be idempotent, recording processed event IDs and ignoring duplicates.
At National Bank of India, Rajesh's transfer committed but crashed before publishing the event. Priya never received an SMS and her statement was stale. After adopting the outbox pattern, every transfer wrote a balance change and a TransferCompleted event in one transaction. Pankaj used Debezium CDC for low-latency relay and made the notification consumer idempotent by event ID.
Remember one sentence: never write to the database and publish to Kafka as two separate steps; write the domain change and an outbox row in one local transaction, let a relay publish with at-least-once delivery, and make consumers idempotent so the effect is exactly-once.
---