Definition
- Race window – two consumers both call
alreadyProcessed(id), both see false, both process, because the gap between check and mark is not atomic. - Crash window – a consumer processes but crashes before
markProcessed(id), so the ID is never marked and redelivery reprocesses. - Fix 1: atomic claim – INSERT with a UNIQUE constraint on event_id so only one consumer wins.
- Fix 2: same transaction – the business change and the dedup record commit together, so a crash cannot separate them.
- Complementary – the unique constraint handles concurrency; the same transaction handles crashes.
Two clerks at National Bank of India both glance at the logbook, both see the instruction number is not logged yet, and both credit Priya, because the glance and the pen-stroke were separate actions with a gap. Separately, one clerk credits Priya but faints before writing the number in the logbook, so when the instruction comes again it looks brand new. The fix is a logbook that physically allows each number to be written only once, and doing the credit and the logbook entry as one inseparable action.
Understanding the Concepts
What does the naive code look like?
The naive dedup has three separate operations: a read, a side effect, and a write, with no atomicity binding them.
package com.bank.consumer;
public class NaiveDedup {
private final ProcessedEventStore store;
private final AccountRepository accountRepo;
// BROKEN: three separate non-atomic operations
public void handle(CreditEvent event) {
// Step 1: READ - check if already processed
if (!store.alreadyProcessed(event.getEventId())) {
// Step 2: SIDE EFFECT - apply the business change
accountRepo.credit(
event.getAccountId(), event.getAmount());
// Step 3: WRITE - mark as processed
store.markProcessed(event.getEventId());
}
}
}
Three separate operations:
Step 1: alreadyProcessed(id) -> READ
Step 2: process() -> SIDE EFFECT
Step 3: markProcessed(id) -> WRITE
Both bugs live in the GAPS between these steps.
How does the concurrency race work?
Two consumers execute alreadyProcessed(id) at nearly the same time. Both reads return false. Both proceed to process.
Concurrency race (two consumers, same event EVT-001):
Consumer A Consumer B
───────── ─────────
alreadyProcessed(EVT-001)? alreadyProcessed(EVT-001)?
-> false -> false
(neither has marked yet!)
credit Priya 50000 credit Priya 50000
markProcessed(EVT-001) markProcessed(EVT-001)
Result: Priya credited 100000 instead of 50000!
The read and the mark are not atomic.
package com.bank.consumer;
// Simulating the race
public class RaceDemo {
public static void main(String[] args) {
// Both threads check at the same time
// Thread A: alreadyProcessed? -> false
// Thread B: alreadyProcessed? -> false (not yet marked!)
// Thread A: credit Priya
// Thread B: credit Priya (duplicate!)
// Thread A: markProcessed
// Thread B: markProcessed
System.out.println("Race condition: Priya credited TWICE");
}
}
How does the crash failure work?
The consumer processes the business change but crashes before marking. On redelivery, the ID is still unmarked.
Crash failure (single consumer, event EVT-001):
Step 1: alreadyProcessed(EVT-001)? -> false
Step 2: credit Priya 50000 -> COMMITTED
*** CRASH ***
Step 3: markProcessed(EVT-001) -> NEVER EXECUTED
Redelivery (broker resends EVT-001):
Step 1: alreadyProcessed(EVT-001)? -> false (never marked!)
Step 2: credit Priya 50000 -> COMMITTED AGAIN
Result: Priya credited 100000 instead of 50000!
Can reordering the lines fix it?
No. Reordering moves the bug but does not eliminate it.
Attempt: mark first, then process
Step 1: markProcessed(EVT-001)
*** CRASH ***
Step 2: process() -> NEVER EXECUTED
Result: event marked but never processed -> money lost!
Attempt: check, mark, then process
Step 1: alreadyProcessed? -> false
Step 2: markProcessed(EVT-001)
*** CRASH ***
Step 3: process() -> NEVER EXECUTED
Result: same problem, just a different gap
The fix is NOT reordering. The fix is ATOMICITY.
Fix 1: How does an atomic INSERT with UNIQUE constraint work?
Instead of reading then writing, attempt a single atomic INSERT. The database guarantees only one insert succeeds.
package com.bank.consumer;
import org.springframework.dao.DuplicateKeyException;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class AtomicClaimConsumer {
private final InboxRepository inbox;
private final AccountRepository accountRepo;
public AtomicClaimConsumer(InboxRepository inbox,
AccountRepository accountRepo) {
this.inbox = inbox;
this.accountRepo = accountRepo;
}
@Transactional
public void handle(CreditEvent event) {
try {
// Atomic claim: INSERT with UNIQUE(event_id)
inbox.insert(event.getEventId());
} catch (DuplicateKeyException e) {
// Another consumer already claimed this event
return; // Skip safely
}
// Only the winner reaches here
accountRepo.credit(
event.getAccountId(), event.getAmount());
}
}
Atomic claim under concurrency:
Consumer A: INSERT EVT-001 -> success (wins)
Consumer B: INSERT EVT-001 -> DuplicateKeyException (loses)
Consumer B: skips processing
Result: Priya credited exactly once
Fix 2: How does a single transaction prevent crash double-processing?
Commit the business change and the inbox record together. Only two outcomes: both present or neither present.
package com.bank.consumer;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class TransactionalInboxConsumer {
private final InboxRepository inbox;
private final AccountRepository accountRepo;
public TransactionalInboxConsumer(InboxRepository inbox,
AccountRepository accountRepo) {
this.inbox = inbox;
this.accountRepo = accountRepo;
}
@Transactional // Both in same transaction
public void handle(CreditEvent event) {
if (inbox.existsByEventId(event.getEventId())) {
return;
}
// Business change
accountRepo.credit(
event.getAccountId(), event.getAmount());
// Dedup record (same transaction)
inbox.save(new InboxEntry(event.getEventId()));
// COMMIT: both together, or neither
// Crash before commit: neither happened -> redelivery safe
// Crash after commit: both present -> duplicate detected
}
}
Same-transaction guarantee:
Crash BEFORE commit: credit not applied, inbox not written
-> redelivery processes from scratch (safe)
Crash AFTER commit: credit applied, inbox written
-> redelivery finds inbox entry, skips (safe)
No in-between state where credit is applied but inbox is missing!
How do the two fixes work together?
| Bug | Root Cause | Fix |
|---|---|---|
| Concurrency race | Check and mark are separate | UNIQUE constraint on event_id |
| Crash double-process | Process and mark are separate | Same database transaction |
| Both combined | Three non-atomic operations | Atomic INSERT + same TX |
What about external side effects?
When the side effect is external (email, API call), you cannot include it in the database transaction. Record the intent in the transaction and perform the external action idempotently afterward.
package com.bank.consumer;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class ConsumerWithExternalEffect {
@Transactional
public void handle(CreditEvent event) {
if (inbox.exists(event.getEventId())) return;
// Database effects in transaction
accountRepo.credit(event.getAccountId(), event.getAmount());
inbox.save(event.getEventId());
// Record intent for external effect
outbox.save(new EmailIntent(
event.getAccountId(), "Credit received"));
}
// Separate process reads outbox, sends email idempotently,
// marks done. If email fails, outbox entry remains for retry.
}
External effects:
Cannot join DB transaction -> record intent in outbox
Outbox processor sends email and marks done
If email service is idempotent -> safe to retry
If not -> accept at-most-once for that effect
Visual Overview
NAIVE (BROKEN):
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Check │ │ Process │ │ Mark │
│ (READ) │ │ (EFFECT) │ │ (WRITE) │
└─────┬────┘ └─────┬────┘ └──────────┘
│ race │ crash
│ window │ window
▼ ▼
FIXED:
┌──────────────────────────────────────────┐
│ Same Transaction │
│ │
│ ┌───────────────────┐ ┌──────────────┐ │
│ │ INSERT inbox │ │ Credit Priya │ │
│ │ UNIQUE(event_id) │ │ 50000.00 │ │
│ │ (atomic claim) │ │ │ │
│ └───────────────────┘ └──────────────┘ │
│ │
│ COMMIT: both or neither │
└──────────────────────────────────────────┘
Concurrency: UNIQUE rejects second INSERT -> only one wins
Crash before COMMIT: nothing happened -> redelivery safe
Crash after COMMIT: both present -> duplicate detected
Key Takeaway
The naive check-then-act dedup fails because its three operations, read, side effect, and write, are not atomic. Under concurrency, two consumers both pass the check because neither has marked yet, so both process. On crash, the consumer processes but the mark is never written, so redelivery reprocesses. Reordering the lines moves the bug but does not eliminate it; the fix is atomicity.
Fix 1 makes the claim atomic with a UNIQUE constraint on event_id, so only one consumer can win the INSERT and the concurrent loser gets a duplicate-key error. Fix 2 commits the business change and the inbox record in the same database transaction, so the only outcomes are both present or neither present, eliminating the crash window. The two fixes are complementary: the unique constraint handles concurrency, the single transaction handles crashes.
At National Bank of India, two consumer instances double-credited Priya because both passed the naive check, and separately a consumer credited her then crashed before marking, and the redelivery credited her again. The team first tried reordering the lines and it still double-processed, which taught everyone that the fix is atomicity, not ordering. The external-effect caveat is that emails and API calls cannot join the database transaction, so you record intent in an outbox and perform the external action idempotently afterward.
Remember one sentence: naive check-then-act dedup double-processes because the check and mark are separate, so fix it with an atomic INSERT under a UNIQUE constraint and commit the mark in the same transaction as the business change.
---