How do you make a message/event consumer safe against at-least-once duplicate delivery | Microservices

Definition

Make the consumer idempotent with an inbox of processed event IDs, a unique constraint, or no-op state transitions, because the real goal is that reprocessing never corrupts state, not literal exactly-once delivery.

At-least-once – brokers guarantee no message is lost but allow duplicates when acks are lost.
Inbox table – stores processed event IDs; new events are checked and recorded in the same transaction as the business change.
Unique constraint – a natural business key rejects duplicate inserts atomically at the database level.
Idempotent state transition – a conditional UPDATE that is a no-op when the state has already moved.
Same-transaction rule – the dedup check and the business change must commit together to survive crashes.
Effectively-once – at-least-once delivery + idempotent processing = each effect applied exactly once.

When the back office at National Bank of India gets a "credit Priya 50000" instruction, it first checks a logbook of instruction numbers it has already handled. If this number is in the logbook, it ignores the instruction. So even if the mail room delivers the same letter twice, Priya is credited only once. The bank does not try to make the postal service deliver exactly once; it makes sure handling a letter twice has the same effect as handling it once.

Understanding the Concepts

Why is exactly-once delivery the wrong target?

Across independent systems, a broker and a consumer, there is always a moment where the consumer has processed but its ack is lost. The broker redelivers. You cannot eliminate this without impossible distributed agreement. So brokers offer at-least-once: no message lost, but duplicates possible. The achievable goal is that processing a duplicate has no additional effect.

TEXT

Why at-least-once allows duplicates:
  1. Consumer receives message
  2. Consumer processes message (credits Priya)
  3. Consumer sends ACK to broker
  4. *** ACK LOST (network issue / crash) ***
  5. Broker: "no ACK received, must redeliver"
  6. Consumer receives same message AGAIN
  7. Without idempotency: credits Priya AGAIN!

  The gap between step 2 and step 3 is irreducible.

How does the inbox technique work?

Keep a table of processed event IDs. Before applying an event, check if its ID is already there. The critical detail: the check and the business change must commit in the same transaction.

JAVA

package com.bank.consumer;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;

@Service
public class IdempotentCreditConsumer {

    private final InboxRepository inbox;
    private final AccountRepository accountRepo;

    public IdempotentCreditConsumer(InboxRepository inbox,
                                     AccountRepository accountRepo) {
        this.inbox = inbox;
        this.accountRepo = accountRepo;
    }

    @KafkaListener(topics = "credit-events")
    @Transactional  // CRITICAL: same transaction for both
    public void onCredit(CreditEvent event) {
        // Check if already processed
        if (inbox.existsByEventId(event.getEventId())) {
            return; // Duplicate: skip
        }

        // Process the business change
        accountRepo.credit(event.getAccountId(),
            event.getAmount());

        // Record in inbox (same transaction!)
        inbox.save(new InboxEntry(event.getEventId()));

        // COMMIT: both credit and inbox entry, or neither
    }
}

SQL

CREATE TABLE inbox (
    event_id VARCHAR(64) PRIMARY KEY,
    processed_at TIMESTAMP DEFAULT NOW()
);

TEXT

First delivery (event_id = "EVT-001"):
  Check inbox: not found
  Credit Priya 50000
  Insert into inbox: "EVT-001"
  COMMIT (both together)

Redelivery (event_id = "EVT-001"):
  Check inbox: found!
  Skip processing
  Priya credited exactly once.

How does a unique constraint work?

When each event produces a record with a naturally unique key, a unique constraint on that key rejects duplicates atomically at the database level.

JAVA

package com.bank.settlement;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.dao.DuplicateKeyException;
import java.math.BigDecimal;

@Service
public class IdempotentSettlementConsumer {

    private final SettlementRepository repo;

    public IdempotentSettlementConsumer(SettlementRepository repo) {
        this.repo = repo;
    }

    @Transactional
    public void onSettlement(SettlementEvent event) {
        try {
            // INSERT with unique constraint on trade_id
            repo.save(new Settlement(
                event.getTradeId(),
                event.getAmount(),
                event.getBeneficiary()));
        } catch (DuplicateKeyException e) {
            // Already processed: the database rejected the duplicate
            // No extra lookup needed!
        }
    }
}

SQL

CREATE TABLE settlements (
    trade_id VARCHAR(64) UNIQUE NOT NULL,
    amount DECIMAL(19,2),
    beneficiary VARCHAR(100),
    created_at TIMESTAMP DEFAULT NOW()
);

TEXT

First delivery (trade_id = "TRD-001"):
  INSERT INTO settlements -> success

Redelivery (trade_id = "TRD-001"):
  INSERT INTO settlements -> DuplicateKeyException
  Caught: skip processing
  Settlement applied exactly once.

How do idempotent state transitions work?

For events that drive a state machine, a conditional UPDATE that only transitions from the expected state is a no-op on duplicates.

JAVA

package com.bank.order;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class IdempotentOrderConsumer {

    private final OrderRepository orderRepo;

    public IdempotentOrderConsumer(OrderRepository orderRepo) {
        this.orderRepo = orderRepo;
    }

    @Transactional
    public void onShipEvent(ShipEvent event) {
        // Conditional update: only if still PLACED
        int updated = orderRepo.updateStatus(
            event.getOrderId(),
            OrderStatus.PLACED,     // from
            OrderStatus.SHIPPED);   // to

        if (updated == 0) {
            // Already SHIPPED or beyond: no-op
            System.out.println("Order " + event.getOrderId() +
                " already shipped, skipping");
        }
    }
}

SQL

UPDATE orders
SET status = 'SHIPPED'
WHERE order_id = 'ORD-001' AND status = 'PLACED';
-- First time:  1 row affected (transitions to SHIPPED)
-- Duplicate:   0 rows affected (already SHIPPED, no-op)

How do the three techniques compare?

| Technique | Best For | Memory | Complexity |
|---|---|---|---|
| Inbox table | General side effects | Grows (prune with TTL) | Medium |
| Unique constraint | Insert-based events | None (DB enforces) | Low |
| State transition | State machine events | None | Low |

Why must the check and mark be in the same transaction?

Separate steps leave a crash window where the business change is applied but the dedup record is not, and a redelivery reprocesses.

TEXT

WRONG (separate steps):
  Step 1: check inbox -> not found
  Step 2: credit Priya 50000 (COMMITTED)
  *** CRASH ***
  Step 3: record in inbox -> NEVER REACHED
  Redelivery: inbox check -> not found -> credits Priya AGAIN!

CORRECT (same transaction):
  BEGIN
    check inbox -> not found
    credit Priya 50000
    record in inbox
  COMMIT
  Crash before COMMIT: neither happened -> redelivery is safe
  Crash after COMMIT: both happened -> redelivery finds inbox entry

How do you handle non-database side effects?

Sending an email or calling an external API cannot join a database transaction. Record the intent in the transaction and perform the external effect idempotently afterward.

JAVA

package com.bank.consumer;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class ConsumerWithExternalEffect {

    @Transactional
    public void process(CreditEvent event) {
        if (inbox.exists(event.getEventId())) return;

        accountRepo.credit(event.getAccountId(), event.getAmount());
        inbox.save(event.getEventId());

        // Record intent to send notification
        outbox.save(new NotificationIntent(
            event.getAccountId(), "Credit received"));
        // COMMIT: all above together
    }

    // Separate process reads outbox and sends notifications
    // If notification fails, outbox entry remains for retry
    // If notification succeeds, outbox entry marked done
}

What are common mistakes?

The most common mistake is checking then acting without atomicity, which races under concurrency and loses the mark on crash.

TEXT

WRONG:
  if (!alreadyProcessed(id)) {  // Check
      process();                  // Act
      markProcessed(id);          // Record
  }
  Three separate steps -> race window + crash window

CORRECT:
  @Transactional
  void handle(event) {
      INSERT inbox(event_id) with UNIQUE constraint  // Atomic claim
      process();                                      // Business change
      // COMMIT: both together
  }

Visual Overview

DIAGRAM

  At-Least-Once Delivery:
  ┌──────────┐   message   ┌──────────┐   ACK    ┌──────────┐
  │ Producer │ ──────────> │  Broker  │ <─────── │ Consumer │
  └──────────┘             │          │  (lost?) └──────────┘
                           │ redeliver│
                           └──────────┘

  Inbox Technique:
  ┌──────────────────────────────────────────────┐
  │               Same Transaction                │
  │                                               │
  │  ┌──────────────┐   ┌────────────────────┐   │
  │  │ Check Inbox  │   │ Business Change    │   │
  │  │ event_id?    │   │ credit Priya 50000 │   │
  │  └──────┬───────┘   └────────────────────┘   │
  │         │                                     │
  │  ┌──────▼───────┐                             │
  │  │ Record in    │                             │
  │  │ Inbox        │                             │
  │  └──────────────┘                             │
  │                                               │
  │  COMMIT: all or nothing                       │
  └──────────────────────────────────────────────┘

Key Takeaway

Making a consumer safe against at-least-once duplicates starts with reframing the problem. Exactly-once delivery is impossible across independent systems because the ack between broker and consumer can always be lost. The achievable goal is that reprocessing a duplicate has no additional effect: effectively-once semantics.

The three techniques are the inbox table (check and record processed event IDs in the same transaction as the business change), a unique constraint on a natural business key (the database atomically rejects duplicates), and idempotent state transitions (conditional updates that are no-ops when the state has already moved). The critical rule across all three is that the dedup check and the business change must commit in the same transaction, because separate steps leave a crash window that causes reprocessing.

At National Bank of India, a consumer double-credited Priya because the ack was lost and the redelivery re-applied the credit. The fix was an inbox table with a unique event_id constraint, where the credit and the inbox entry commit in the same transaction. On redelivery, the inbox check finds the entry and skips. For settlements, a unique trade_id constraint on the settlements table rejects duplicates at the database level without any extra lookup.

Remember one sentence: make the consumer idempotent with an inbox committed in the same transaction as the business change, a unique constraint, or a conditional state transition, because the goal is that reprocessing never corrupts state, not literal exactly-once delivery.

---