Definition
- Send-then-mark – the relay publishes to Kafka first, then marks the outbox row published; a crash between those steps leaves the row unpublished, causing re-delivery.
- At-least-once – every event is delivered one or more times, never zero; duplicates are expected.
- Idempotent consumer – processing the same event twice has the same effect as once, typically via a processed-events table keyed by event ID.
- Retention policy – a scheduled job deletes published rows older than a window (e.g., 7 days); for high volume, partition the table by day and drop old partitions.
- Batch and throttle – deletes should be batched to avoid long locks during business hours.
The messenger at National Bank of India grabs a transfer slip from the outbox drawer, hands it to the Kafka courier, and is supposed to stamp it "SENT." One day the messenger faints before stamping. When he wakes up, the slip still looks unsent, so he hands it to the courier again. That is fine as long as the receiving desk is smart: when it gets a slip it has already seen (same slip number), it throws the duplicate away. And because the drawer fills up, a clerk shreds all stamped slips nightly so the drawer never overflows.
Understanding the Concepts
What happens when the relay crashes between send and mark?
The relay does three things in order: select unpublished rows, publish each to Kafka, mark each published. Publishing and marking are two different systems, so there is no atomic way to do both. If the relay crashes after Kafka send but before the DB mark, the row stays unpublished. On restart the relay re-selects it and publishes again.
Why send-first, not mark-first?
If you mark first and crash before sending, the event is lost entirely: at-most-once, which is unacceptable for money. Send-first, mark-second accepts duplicates, which consumers can handle.
// Send-first, mark-second: at-least-once (safe)
kafka.send(topic, event); // if crash HERE...
outboxRepo.markPublished(event.getId()); // ...row stays unpublished → re-sent on restart
// Mark-first, send-second: at-most-once (DANGEROUS)
outboxRepo.markPublished(event.getId()); // if crash HERE...
kafka.send(topic, event); // ...event LOST forever!
How do consumers achieve idempotency?
There are four approaches.
1. Processed-events table
package com.bank.consumer;
import org.springframework.transaction.annotation.Transactional;
public class TransferNotificationConsumer {
private final ProcessedEventRepository processedRepo;
private final NotificationService notifier;
@Transactional
public void handle(TransferEvent event) {
if (processedRepo.existsById(event.getEventId())) {
return; // duplicate – skip
}
notifier.sendSms(event.getCustomerId(), "Transfer complete: " + event.getAmount());
processedRepo.save(event.getEventId()); // record in SAME transaction
}
}
2. Natural idempotency
If the operation is "set balance to X" rather than "add X," repeating it is harmless.
3. Upsert keyed by business ID
A repeated insert becomes a no-op update.
4. Unique constraint on event ID
The second insert throws a constraint violation, which the consumer swallows.
Why must the dedup check and business write be in one transaction?
A crash between "wrote the business change" and "recorded the event ID" would re-process on restart. Putting both in one transaction closes that gap.
How do you prevent the outbox table from growing?
-- Retention job: delete published rows older than 7 days
DELETE FROM outbox
WHERE published = TRUE
AND created_at < NOW() - INTERVAL '7 days'
LIMIT 10000; -- batch to avoid long locks
For high volume, partition the outbox table by day and drop old partitions, which is far cheaper than row-by-row deletes.
What should you monitor?
Alert if the count of unpublished rows (the relay backlog) grows, because that means the relay or Kafka is stuck.
What are common mistakes?
// WRONG: mark-first, then send → event can be lost
outboxRepo.markPublished(id);
kafka.send(topic, event); // crash here = event lost forever
// CORRECT: send-first, then mark → duplicate is safe
kafka.send(topic, event);
outboxRepo.markPublished(id); // crash here = re-sent, consumer deduplicates
What should a senior engineer know?
- At-least-once delivery pushes correctness responsibility onto the consumer: the relay promises to deliver at least once, the consumer promises to be idempotent, and together they give effectively-once business behavior.
- Debezium's CDC can be configured to delete outbox rows after routing, keeping the table tiny.
- Deletes should be batched and throttled (e.g., 10,000 rows at a time with a pause) to avoid I/O spikes.
- Monitoring relay backlog is the single most important health signal for the outbox.
Visual Overview
Relay Crash Scenario
1. Relay reads outbox row (event-42, unpublished)
2. Relay sends to Kafka ✓ sent
3. Relay crashes BEFORE marking published ✗
4. On restart, row still says unpublished
5. Relay sends event-42 AGAIN → duplicate
Consumer Idempotency
┌──────────┐ event-42 ┌───────────────────┐
│ Kafka │ ─────────► │ Consumer │
└──────────┘ │ 1. check eventId │
│ 2. already seen? │
│ YES → skip │
│ NO → process │
│ 3. record eventId │
└───────────────────┘
Outbox Retention
┌────────────────────────────────────────┐
│ outbox table │
│ [unpublished] → relay picks up │
│ [published, age < 7d] → kept for debug │
│ [published, age > 7d] → DELETE batch │
└────────────────────────────────────────┘
Key Takeaway
In an outbox design the relay reads an unpublished row, publishes it to Kafka, and then marks it published. Those last two steps hit two different systems, so there is no atomic way to do both. If the relay crashes after the Kafka send but before the database mark, the row stays unpublished and the relay re-sends it on restart. The design deliberately orders it send-first, mark-second, because mark-first, send-second risks losing the event entirely, and silently losing a banking event is far worse than delivering it twice.
The direct consequence is that duplicates are guaranteed, so downstream consumers must be idempotent. In the banking notification service, Rajesh's transfer might trigger two identical SMS messages unless the consumer checks a processed-events table keyed by event ID and skips duplicates. The dedup check and the business write must be in one transaction to avoid a gap where the consumer crashes between the two.
The second operational concern is that every committed transfer inserts an outbox row. Without cleanup the table grows unboundedly, slowing the relay and filling disk. A retention job deletes published rows older than a window, keeping recent ones for debugging. For high volume, partitioning by day and dropping old partitions is far cheaper than row-by-row deletes.
Remember one sentence: send-then-mark makes the relay re-send on crash, so consumers must deduplicate by event ID, and a retention job must delete published rows so the outbox stays bounded.
---