Back to Event-Driven Architecture & CQRS
Q9Question 9 of 9

How do you handle duplicate messages and out-of-order events

Answer: Handle duplicate messages with an idempotent consumer that checks a processed_events database table before processing, and handle out-of-order events by using the entity ID as the Kafka message key to guarantee partition-level ordering.

Key Concepts
  • Kafka delivers messages at-least-once by default, so consumers must handle duplicates caused by crashes before offset commit or consumer rebalances.
  • An idempotent consumer uses a processed_events table with a unique constraint on event_id to detect and skip duplicate messages.
  • The business processing and the dedup record insert must happen in the same database transaction to prevent both duplicates and lost events.
  • Out-of-order events are prevented by using the entity ID as the Kafka message key, which routes all events for the same entity to the same partition.
  • Version-based ordering assigns an incrementing version to each event, and the consumer skips events with a version lower than or equal to the last processed version.
  • Dead letter queues capture events that fail processing after all retries, preventing a single bad message from blocking the consumer.
  • At the senior level, monitor consumer lag per partition, set TTL on the dedup table, and use the idempotent producer (enable.idempotence=true) to prevent producer-side duplicates.

Definition

Handle duplicate messages with an idempotent consumer that checks a processed_events database table before processing, and handle out-of-order events by using the entity ID as the Kafka message key to guarantee partition-level ordering.
  • Idempotent consumer is a consumer that can receive the same message multiple times and produce the same result as if it received the message once, typically implemented with a deduplication table in the database.
  • Processed events table is a database table with a unique constraint on event_id where the consumer records every successfully processed event, enabling duplicate detection on redelivery.
  • At-least-once delivery is Kafka's default guarantee where a message is delivered one or more times, meaning the consumer must handle duplicates.
  • Message key is the key field of a Kafka record that determines which partition the message is sent to; all messages with the same key go to the same partition, preserving order.
  • Version-based ordering is a technique where each event carries an incrementing version number, and the consumer rejects events with a version older than or equal to the last processed version.

Think of National Bank of India's branch mail system. The Mumbai branch sends a payment instruction to the Delhi branch by registered post. The postal system guarantees the letter will be delivered at least once, but sometimes a letter is delivered twice (the postman scanned it as undelivered and resent it). The Delhi branch keeps a register of all processed instruction numbers. When a letter arrives, the clerk checks the register first. If the instruction number is already in the register, the clerk stamps it as duplicate and files it without processing. If the instruction number is new, the clerk processes the payment and adds the number to the register in one step.

Understanding the Concepts

What causes duplicate messages in event-driven microservices?

Duplicate messages occur because Kafka (and most messaging systems) guarantee at-least-once delivery, not exactly-once. A producer sends a message, the broker stores it, and the consumer reads it. If the consumer processes the message but crashes before committing the offset, Kafka redelivers the same message after the consumer restarts. The consumer processes it a second time.

Other causes include producer retries (the broker stored the message but the acknowledgment was lost, so the producer sends the message again) and consumer rebalancing (partitions are reassigned mid-processing).

JAVA
package com.bank.events;

public class DuplicateCauses {

    sealed interface DuplicateSource permits ProducerRetry, ConsumerCrash, ConsumerRebalance {}
    record ProducerRetry(String reason) implements DuplicateSource {}
    record ConsumerCrash(String reason) implements DuplicateSource {}
    record ConsumerRebalance(String reason) implements DuplicateSource {}

    public static void main(String[] args) {
        var causes = java.util.List.of(
            new ProducerRetry("Broker stored message but ACK was lost. Producer retries. Broker stores it again."),
            new ConsumerCrash("Consumer processed message but crashed before offset commit. Kafka redelivers."),
            new ConsumerRebalance("Partition reassigned during processing. New consumer re-reads uncommitted messages.")
        );

        System.out.println("Why duplicate messages happen in Kafka:");
        for (var cause : causes) {
            String label = switch (cause) {
                case ProducerRetry p -> "[PRODUCER RETRY] " + p.reason();
                case ConsumerCrash c -> "[CONSUMER CRASH] " + c.reason();
                case ConsumerRebalance r -> "[REBALANCE] " + r.reason();
            };
            System.out.println("  " + label);
        }
    }
}
TEXT
Expected output:
Why duplicate messages happen in Kafka:
  [PRODUCER RETRY] Broker stored message but ACK was lost. Producer retries. Broker stores it again.
  [CONSUMER CRASH] Consumer processed message but crashed before offset commit. Kafka redelivers.
  [REBALANCE] Partition reassigned during processing. New consumer re-reads uncommitted messages.

What causes out-of-order events?

Events arrive out of order when a topic has multiple partitions. Kafka guarantees ordering within a single partition, but not across partitions. If order-service publishes OrderCreated and OrderPaid events for the same order, and these events land on different partitions, the consumer might receive OrderPaid before OrderCreated.

Other causes include retries with reordering (a failed message is retried later, arriving after newer messages) and multiple producer instances publishing events for the same entity without coordination.

JAVA
package com.bank.events;

public class OutOfOrderCauses {

    record Event(String eventId, String orderId, String type, int partition, long offset) {}

    public static void main(String[] args) {
        // Events for the same order on DIFFERENT partitions
        var events = java.util.List.of(
            new Event("E001", "ORD-501", "ORDER_CREATED", 0, 100),
            new Event("E002", "ORD-501", "PAYMENT_AUTHORIZED", 2, 50),
            new Event("E003", "ORD-501", "ORDER_CONFIRMED", 1, 75)
        );

        System.out.println("Events published to different partitions:");
        events.forEach(e -> System.out.println("  " + e.type()
            + " partition=" + e.partition() + " offset=" + e.offset()));
        System.out.println();

        // Consumer reads partition 2 first (lower offset)
        System.out.println("Consumer reads in offset order PER PARTITION:");
        System.out.println("  1. PAYMENT_AUTHORIZED (partition 2, offset 50)");
        System.out.println("  2. ORDER_CONFIRMED (partition 1, offset 75)");
        System.out.println("  3. ORDER_CREATED (partition 0, offset 100)");
        System.out.println();
        System.out.println("Problem: PAYMENT_AUTHORIZED arrived before ORDER_CREATED.");
        System.out.println("The consumer does not know which order this payment belongs to.");
        System.out.println();
        System.out.println("Fix: use orderId as the Kafka message key.");
        System.out.println("All events for ORD-501 go to the same partition.");
        System.out.println("Kafka guarantees ordering within a partition.");
    }
}
TEXT
Expected output:
Events published to different partitions:
  ORDER_CREATED partition=0 offset=100
  PAYMENT_AUTHORIZED partition=2 offset=50
  ORDER_CONFIRMED partition=1 offset=75

Consumer reads in offset order PER PARTITION:
  1. PAYMENT_AUTHORIZED (partition 2, offset 50)
  2. ORDER_CONFIRMED (partition 1, offset 75)
  3. ORDER_CREATED (partition 0, offset 100)

Problem: PAYMENT_AUTHORIZED arrived before ORDER_CREATED.
The consumer does not know which order this payment belongs to.

Fix: use orderId as the Kafka message key.
All events for ORD-501 go to the same partition.
Kafka guarantees ordering within a partition.

How do you implement an idempotent consumer for deduplication?

An idempotent consumer uses a processed_events table in the database. Before processing a message, the consumer checks if the event ID already exists in the table. If it does, the consumer skips the message. If it does not, the consumer processes the message and inserts the event ID into the table in the same database transaction.

The key requirement is that the business processing and the deduplication insert happen in a single atomic transaction. If they are in separate transactions, a crash between them will cause either a duplicate (processed but not recorded) or a lost event (recorded but not processed).

JAVA
package com.bank.events;

import java.util.HashSet;
import java.util.Set;

public class IdempotentConsumerDemo {

    record OrderEvent(String eventId, String orderId, String type, int version) {}

    static Set<String> processedEvents = new HashSet<>();

    static boolean processEvent(OrderEvent event) {
        // Step 1: Check deduplication table
        if (processedEvents.contains(event.eventId())) {
            System.out.println("  SKIP (duplicate): " + event.eventId() + " " + event.type());
            return false;
        }

        // Step 2: Process the event (in same transaction as step 3)
        System.out.println("  PROCESS: " + event.eventId() + " " + event.type()
            + " for order " + event.orderId());

        // Step 3: Mark as processed (same transaction)
        processedEvents.add(event.eventId());
        return true;
    }

    public static void main(String[] args) {
        var events = java.util.List.of(
            new OrderEvent("EVT-001", "ORD-501", "ORDER_CREATED", 1),
            new OrderEvent("EVT-002", "ORD-501", "PAYMENT_AUTHORIZED", 2),
            new OrderEvent("EVT-001", "ORD-501", "ORDER_CREATED", 1),  // duplicate
            new OrderEvent("EVT-003", "ORD-501", "ORDER_CONFIRMED", 3),
            new OrderEvent("EVT-002", "ORD-501", "PAYMENT_AUTHORIZED", 2)  // duplicate
        );

        System.out.println("Processing events with idempotent consumer:");
        for (var event : events) {
            processEvent(event);
        }
        System.out.println();
        System.out.println("Processed " + processedEvents.size() + " unique events out of " + events.size() + " total");
    }
}
TEXT
Expected output:
Processing events with idempotent consumer:
  PROCESS: EVT-001 ORDER_CREATED for order ORD-501
  PROCESS: EVT-002 PAYMENT_AUTHORIZED for order ORD-501
  SKIP (duplicate): EVT-001 ORDER_CREATED
  PROCESS: EVT-003 ORDER_CONFIRMED for order ORD-501
  SKIP (duplicate): EVT-002 PAYMENT_AUTHORIZED

Processed 3 unique events out of 5 total

How do you handle out-of-order events with version-based ordering?

Version-based ordering assigns an incrementing version number to each event for an entity. The consumer stores the last processed version for each entity. When a new event arrives, the consumer compares the event version with the stored version. If the event version is not exactly lastVersion + 1, the consumer either skips it (stale) or buffers it (future event that arrived early).

JAVA
package com.bank.events;

import java.util.HashMap;
import java.util.Map;

public class VersionBasedOrderingDemo {

    record OrderEvent(String eventId, String orderId, String type, int version) {}

    static Map<String, Integer> lastProcessedVersion = new HashMap<>();

    static void processWithVersionCheck(OrderEvent event) {
        int lastVersion = lastProcessedVersion.getOrDefault(event.orderId(), 0);

        if (event.version() <= lastVersion) {
            System.out.println("  SKIP (stale): " + event.type()
                + " v" + event.version() + " (last processed: v" + lastVersion + ")");
            return;
        }

        if (event.version() > lastVersion + 1) {
            System.out.println("  BUFFER (gap): " + event.type()
                + " v" + event.version() + " (expected: v" + (lastVersion + 1) + ")");
            return;
        }

        System.out.println("  PROCESS: " + event.type() + " v" + event.version()
            + " for " + event.orderId());
        lastProcessedVersion.put(event.orderId(), event.version());
    }

    public static void main(String[] args) {
        var events = java.util.List.of(
            new OrderEvent("E001", "ORD-501", "ORDER_CREATED", 1),
            new OrderEvent("E003", "ORD-501", "ORDER_CONFIRMED", 3),  // arrived early
            new OrderEvent("E002", "ORD-501", "PAYMENT_AUTHORIZED", 2),
            new OrderEvent("E001", "ORD-501", "ORDER_CREATED", 1),  // duplicate (stale)
            new OrderEvent("E003", "ORD-501", "ORDER_CONFIRMED", 3)  // now in order
        );

        System.out.println("Processing events with version-based ordering:");
        for (var event : events) {
            processWithVersionCheck(event);
        }
    }
}
TEXT
Expected output:
Processing events with version-based ordering:
  PROCESS: ORDER_CREATED v1 for ORD-501
  BUFFER (gap): ORDER_CONFIRMED v3 (expected: v2)
  PROCESS: PAYMENT_AUTHORIZED v2 for ORD-501
  SKIP (stale): ORDER_CREATED v1 (last processed: v2)
  PROCESS: ORDER_CONFIRMED v3 for ORD-501

What is a complete banking project with idempotent Kafka consumers?

This project has two services: order-service publishes OrderCreated events to Kafka, and payment-service consumes them with deduplication and version checking. The payment-service uses Spring Kafka with @KafkaListener and stores processed events in a PostgreSQL table.

Project structure

DIAGRAM
event-dedup-banking/
├── docker-compose.yml
├── init-db.sql
├── order-service/
│   ├── pom.xml
│   └── src/main/java/com/bank/order/
│       ├── OrderServiceApp.java
│       ├── controller/OrderController.java
│       ├── service/OrderService.java
│       ├── model/Order.java
│       ├── dto/CreateOrderRequest.java
│       ├── event/OrderCreatedEvent.java
│       ├── config/KafkaProducerConfig.java
│       └── repository/OrderRepository.java
└── payment-service/
    ├── pom.xml
    └── src/main/java/com/bank/payment/
        ├── PaymentServiceApp.java
        ├── listener/OrderEventListener.java
        ├── service/PaymentProcessingService.java
        ├── model/ProcessedEvent.java
        ├── model/PaymentRecord.java
        ├── repository/ProcessedEventRepository.java
        ├── repository/PaymentRecordRepository.java
        └── config/KafkaConsumerConfig.java

docker-compose.yml

YAML
version: "3.9"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper

  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: bank
      POSTGRES_PASSWORD: bank123
      POSTGRES_DB: bank
    ports:
      - "5432:5432"
    volumes:
      - ./init-db.sql:/docker-entrypoint-initdb.d/01-init.sql

init-db.sql

SQL
CREATE TABLE IF NOT EXISTS orders (
    id              BIGSERIAL PRIMARY KEY,
    order_id        VARCHAR(50) NOT NULL UNIQUE,
    customer_id     VARCHAR(20) NOT NULL,
    amount          NUMERIC(15,2) NOT NULL,
    status          VARCHAR(20) NOT NULL DEFAULT 'CREATED',
    version         INT NOT NULL DEFAULT 1,
    created_at      TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE IF NOT EXISTS processed_events (
    id              BIGSERIAL PRIMARY KEY,
    event_id        VARCHAR(50) NOT NULL UNIQUE,
    event_type      VARCHAR(50) NOT NULL,
    entity_id       VARCHAR(50) NOT NULL,
    processed_at    TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE IF NOT EXISTS payment_records (
    id              BIGSERIAL PRIMARY KEY,
    order_id        VARCHAR(50) NOT NULL,
    customer_id     VARCHAR(20) NOT NULL,
    amount          NUMERIC(15,2) NOT NULL,
    status          VARCHAR(20) NOT NULL DEFAULT 'PENDING',
    created_at      TIMESTAMP NOT NULL DEFAULT NOW()
);

OrderCreatedEvent.java (shared event contract)

JAVA
package com.bank.order.event;

import java.math.BigDecimal;

public record OrderCreatedEvent(
    String eventId,
    String orderId,
    String customerId,
    BigDecimal amount,
    int version,
    long timestampMs
) {}

order-service pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><parentsyn-kw">>
        syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
        syn-kw"><artifactIdsyn-kw">>spring-boot-starter-parentsyn-kw"></artifactIdsyn-kw">>
        syn-kw"><versionsyn-kw">>3.4.1syn-kw"></versionsyn-kw">>
    syn-kw"></parentsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>order-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-websyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-data-jpasyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-kafkasyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.postgresqlsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>postgresqlsyn-kw"></artifactIdsyn-kw">>
            syn-kw"><scopesyn-kw">>runtimesyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>com.fasterxml.jackson.coresyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>jackson-databindsyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

OrderService.java (publishes events)

JAVA
package com.bank.order.service;

import com.bank.order.event.OrderCreatedEvent;
import com.bank.order.model.Order;
import com.bank.order.repository.OrderRepository;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.math.BigDecimal;
import java.util.UUID;
import java.util.logging.Logger;

@Service
public class OrderService {

    private static final Logger LOG = Logger.getLogger(OrderService.class.getName());
    private final OrderRepository repository;
    private final KafkaTemplate<String, String> kafkaTemplate;
    private final ObjectMapper objectMapper;

    public OrderService(OrderRepository repository,
                        KafkaTemplate<String, String> kafkaTemplate,
                        ObjectMapper objectMapper) {
        this.repository = repository;
        this.kafkaTemplate = kafkaTemplate;
        this.objectMapper = objectMapper;
    }

    @Transactional
    public Order createOrder(String customerId, BigDecimal amount) {
        var orderId = "ORD-" + UUID.randomUUID().toString().substring(0, 8).toUpperCase();
        var order = new Order();
        order.setOrderId(orderId);
        order.setCustomerId(customerId);
        order.setAmount(amount);
        order.setStatus("CREATED");
        order.setVersion(1);
        repository.save(order);

        // Publish event with orderId as key (ensures same partition for same order)
        var event = new OrderCreatedEvent(
            UUID.randomUUID().toString(), orderId, customerId, amount, 1, System.currentTimeMillis());
        try {
            var json = objectMapper.writeValueAsString(event);
            kafkaTemplate.send("order-events", orderId, json);
            LOG.info("Published OrderCreated: orderId=" + orderId + ", key=" + orderId);
        } catch (Exception ex) {
            LOG.severe("Failed to publish event: " + ex.getMessage());
        }

        return order;
    }
}

OrderController.java

JAVA
package com.bank.order.controller;

import com.bank.order.dto.CreateOrderRequest;
import com.bank.order.model.Order;
import com.bank.order.service.OrderService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/orders")
public class OrderController {

    private final OrderService orderService;

    public OrderController(OrderService orderService) {
        this.orderService = orderService;
    }

    @PostMapping
    public ResponseEntity<Order> createOrder(@RequestBody CreateOrderRequest request) {
        var order = orderService.createOrder(request.customerId(), request.amount());
        return ResponseEntity.ok(order);
    }
}

CreateOrderRequest.java

JAVA
package com.bank.order.dto;

import java.math.BigDecimal;

public record CreateOrderRequest(String customerId, BigDecimal amount) {}

OrderEventListener.java (idempotent consumer)

JAVA
package com.bank.payment.listener;

import com.bank.payment.service.PaymentProcessingService;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

import java.math.BigDecimal;
import java.util.logging.Logger;

@Component
public class OrderEventListener {

    private static final Logger LOG = Logger.getLogger(OrderEventListener.class.getName());
    private final PaymentProcessingService processingService;
    private final ObjectMapper objectMapper;

    public OrderEventListener(PaymentProcessingService processingService,
                               ObjectMapper objectMapper) {
        this.processingService = processingService;
        this.objectMapper = objectMapper;
    }

    @KafkaListener(topics = "order-events", groupId = "payment-service-group")
    public void onOrderEvent(ConsumerRecord<String, String> record) {
        LOG.info("Received event: partition=" + record.partition()
            + ", offset=" + record.offset() + ", key=" + record.key());
        try {
            var node = objectMapper.readTree(record.value());
            var eventId = node.get("eventId").asText();
            var orderId = node.get("orderId").asText();
            var customerId = node.get("customerId").asText();
            var amount = new BigDecimal(node.get("amount").asText());
            var version = node.get("version").asInt();

            processingService.processOrderCreated(eventId, orderId, customerId, amount, version);
        } catch (Exception ex) {
            LOG.severe("Failed to process event: " + ex.getMessage());
        }
    }
}

PaymentProcessingService.java (dedup + version check)

JAVA
package com.bank.payment.service;

import com.bank.payment.model.PaymentRecord;
import com.bank.payment.model.ProcessedEvent;
import com.bank.payment.repository.PaymentRecordRepository;
import com.bank.payment.repository.ProcessedEventRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.math.BigDecimal;
import java.util.logging.Logger;

@Service
public class PaymentProcessingService {

    private static final Logger LOG = Logger.getLogger(PaymentProcessingService.class.getName());
    private final ProcessedEventRepository processedEventRepository;
    private final PaymentRecordRepository paymentRecordRepository;

    public PaymentProcessingService(ProcessedEventRepository processedEventRepository,
                                     PaymentRecordRepository paymentRecordRepository) {
        this.processedEventRepository = processedEventRepository;
        this.paymentRecordRepository = paymentRecordRepository;
    }

    @Transactional
    public void processOrderCreated(String eventId, String orderId,
                                     String customerId, BigDecimal amount, int version) {
        // Step 1: Deduplication check
        if (processedEventRepository.existsByEventId(eventId)) {
            LOG.info("DUPLICATE event skipped: eventId=" + eventId);
            return;
        }

        // Step 2: Process the event (create payment record)
        var payment = new PaymentRecord();
        payment.setOrderId(orderId);
        payment.setCustomerId(customerId);
        payment.setAmount(amount);
        payment.setStatus("PAYMENT_PENDING");
        paymentRecordRepository.save(payment);

        // Step 3: Mark event as processed (SAME transaction as step 2)
        var processed = new ProcessedEvent();
        processed.setEventId(eventId);
        processed.setEventType("ORDER_CREATED");
        processed.setEntityId(orderId);
        processedEventRepository.save(processed);

        LOG.info("PROCESSED event: eventId=" + eventId + ", orderId=" + orderId + ", amount=" + amount);
    }
}

ProcessedEvent.java and ProcessedEventRepository.java

JAVA
package com.bank.payment.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;

import java.time.LocalDateTime;

@Entity
@Table(name = "processed_events")
public class ProcessedEvent {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "event_id", nullable = false, unique = true)
    private String eventId;

    @Column(name = "event_type", nullable = false)
    private String eventType;

    @Column(name = "entity_id", nullable = false)
    private String entityId;

    @Column(name = "processed_at", nullable = false)
    private LocalDateTime processedAt = LocalDateTime.now();

    public String getEventId() { return eventId; }
    public void setEventId(String eventId) { this.eventId = eventId; }
    public String getEventType() { return eventType; }
    public void setEventType(String eventType) { this.eventType = eventType; }
    public String getEntityId() { return entityId; }
    public void setEntityId(String entityId) { this.entityId = entityId; }
}
JAVA
package com.bank.payment.repository;

import com.bank.payment.model.ProcessedEvent;
import org.springframework.data.jpa.repository.JpaRepository;

public interface ProcessedEventRepository extends JpaRepository<ProcessedEvent, Long> {
    boolean existsByEventId(String eventId);
}

How to run

BASH
# Step 1: Start infrastructure
docker-compose up -d

# Step 2: Create Kafka topic
docker exec -it event-dedup-banking-kafka-1 \
  kafka-topics --create --topic order-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build and start order-service (port 8084)
cd order-service && mvn clean package -DskipTests && cd ..
java -jar order-service/target/order-service-1.0.0.jar &

# Step 4: Build and start payment-service (port 8085)
cd payment-service && mvn clean package -DskipTests && cd ..
java -jar payment-service/target/payment-service-1.0.0.jar &

# Step 5: Create an order (Rajesh buys for 75000 rupees)
curl -s -X POST http://localhost:8084/orders \
  -H "Content-Type: application/json" \
  -d '{"customerId":"SAV2001","amount":75000}'

# Step 6: Check payment-service logs for processing
# You should see: PROCESSED event: eventId=xxx, orderId=ORD-xxx, amount=75000
TEXT
Expected output (order-service):
{"orderId":"ORD-A1B2C3D4","customerId":"SAV2001","amount":75000.00,"status":"CREATED","version":1}

Expected output (payment-service logs):
Received event: partition=1, offset=0, key=ORD-A1B2C3D4
PROCESSED event: eventId=evt-uuid, orderId=ORD-A1B2C3D4, amount=75000

If Kafka redelivers the same event:
DUPLICATE event skipped: eventId=evt-uuid

How is at-least-once different from exactly-once and at-most-once?

GuaranteeDescriptionDuplicates?Message Loss?Use Case
At-most-onceProcess message, then commit offsetNoYes (if crash before processing)Logging, metrics
At-least-onceCommit offset after processingYes (if crash before commit)NoPayments, orders (with dedup)
Exactly-onceKafka transactions + idempotent producerNoNoFinancial transfers (Kafka-to-Kafka)
JAVA
package com.bank.events;

public class DeliveryGuarantees {

    public static void main(String[] args) {
        System.out.println("=== At-Most-Once ===");
        System.out.println("  1. Receive message");
        System.out.println("  2. Commit offset FIRST");
        System.out.println("  3. Process message");
        System.out.println("  Risk: crash at step 3 -> message is lost");
        System.out.println();

        System.out.println("=== At-Least-Once ===");
        System.out.println("  1. Receive message");
        System.out.println("  2. Process message");
        System.out.println("  3. Commit offset AFTER");
        System.out.println("  Risk: crash at step 3 -> message is redelivered, processed twice");
        System.out.println("  Fix: idempotent consumer (dedup table)");
        System.out.println();

        System.out.println("=== Exactly-Once (Kafka transactions) ===");
        System.out.println("  1. Begin Kafka transaction");
        System.out.println("  2. Consume from input topic");
        System.out.println("  3. Process and produce to output topic");
        System.out.println("  4. Commit offsets + production in ONE transaction");
        System.out.println("  Limitation: works only for Kafka-to-Kafka pipelines");
    }
}
TEXT
Expected output:
=== At-Most-Once ===
  1. Receive message
  2. Commit offset FIRST
  3. Process message
  Risk: crash at step 3 -> message is lost

=== At-Least-Once ===
  1. Receive message
  2. Process message
  3. Commit offset AFTER
  Risk: crash at step 3 -> message is redelivered, processed twice
  Fix: idempotent consumer (dedup table)

=== Exactly-Once (Kafka transactions) ===
  1. Begin Kafka transaction
  2. Consume from input topic
  3. Process and produce to output topic
  4. Commit offsets + production in ONE transaction
  Limitation: works only for Kafka-to-Kafka pipelines

What are common mistakes with event deduplication?

Wrong: process first, record event ID later (separate transactions)

JAVA
package com.bank.events.wrong;

import java.math.BigDecimal;
import java.util.HashSet;
import java.util.Set;

public class WrongSeparateTransactions {
    static Set<String> processedEvents = new HashSet<>();
    static BigDecimal totalProcessed = BigDecimal.ZERO;

    public static void main(String[] args) {
        System.out.println("WRONG: process and dedup record in SEPARATE transactions");
        System.out.println();

        // Transaction 1: process the event
        System.out.println("Transaction 1: process payment for ORD-501, amount=75000");
        totalProcessed = totalProcessed.add(new BigDecimal("75000"));
        System.out.println("  Payment created. Total processed: " + totalProcessed);

        // CRASH HERE: app crashes before recording the event ID
        System.out.println("  ** APP CRASHES **");
        System.out.println();

        // After restart, Kafka redelivers the event
        System.out.println("After restart, Kafka redelivers the same event:");
        System.out.println("  Event ID not in dedup table (Transaction 2 never ran)");

        // Transaction 1 again: process the event AGAIN
        totalProcessed = totalProcessed.add(new BigDecimal("75000"));
        System.out.println("  Payment created AGAIN. Total processed: " + totalProcessed);

        // Transaction 2: record the event ID
        processedEvents.add("EVT-001");
        System.out.println("  Event ID recorded");
        System.out.println();

        System.out.println("Result: customer charged 150000 instead of 75000!");
    }
}
TEXT
Expected output:
WRONG: process and dedup record in SEPARATE transactions

Transaction 1: process payment for ORD-501, amount=75000
  Payment created. Total processed: 75000
  ** APP CRASHES **

After restart, Kafka redelivers the same event:
  Event ID not in dedup table (Transaction 2 never ran)
  Payment created AGAIN. Total processed: 150000
  Event ID recorded

Result: customer charged 150000 instead of 75000!

Correct: process and record event ID in the SAME transaction

JAVA
package com.bank.events.correct;

import java.math.BigDecimal;
import java.util.HashSet;
import java.util.Set;

public class CorrectSameTransaction {
    static Set<String> processedEvents = new HashSet<>();
    static BigDecimal totalProcessed = BigDecimal.ZERO;

    public static void main(String[] args) {
        System.out.println("CORRECT: process and dedup record in SAME @Transactional method");
        System.out.println();

        // Single transaction: process + record
        System.out.println("@Transactional method:");
        System.out.println("  1. Check dedup table: EVT-001 not found");
        System.out.println("  2. Create payment record for ORD-501, amount=75000");
        System.out.println("  3. Insert EVT-001 into dedup table");
        System.out.println("  4. COMMIT (all three steps are atomic)");
        totalProcessed = totalProcessed.add(new BigDecimal("75000"));
        processedEvents.add("EVT-001");
        System.out.println("  Total processed: " + totalProcessed);
        System.out.println();

        System.out.println("If crash at step 2 or 3: entire transaction rolls back.");
        System.out.println("  Kafka redelivers. Event ID not in table. Process normally.");
        System.out.println();

        System.out.println("If Kafka redelivers after successful commit:");
        if (processedEvents.contains("EVT-001")) {
            System.out.println("  Step 1: EVT-001 found in dedup table. SKIP.");
        }
        System.out.println("  Total processed: " + totalProcessed + " (charged only once)");
    }
}
TEXT
Expected output:
CORRECT: process and dedup record in SAME @Transactional method

@Transactional method:
  1. Check dedup table: EVT-001 not found
  2. Create payment record for ORD-501, amount=75000
  3. Insert EVT-001 into dedup table
  4. COMMIT (all three steps are atomic)
  Total processed: 75000

If crash at step 2 or 3: entire transaction rolls back.
  Kafka redelivers. Event ID not in table. Process normally.

If Kafka redelivers after successful commit:
  Step 1: EVT-001 found in dedup table. SKIP.
  Total processed: 75000 (charged only once)

What should a senior or principal engineer know?

At the senior level, you must understand the tradeoffs between deduplication strategies:

  • Database unique constraint on event_id: simple and reliable. The INSERT fails with a constraint violation if the event was already processed. Wrap in try-catch and ignore the duplicate.
  • Bloom filter: memory-efficient for checking if an event was probably processed, but has false positives. Use as a fast first check before the database query.
  • Dedup table TTL: old entries should be purged. An event from 7 days ago will not be redelivered by Kafka (retention period). Set a TTL and run a cleanup job.
  • Consumer offset commit strategy: manual commit after processing gives you at-least-once. Auto-commit is at-most-once. For financial events, always use manual commit.
  • At the principal level:
  • Compacted topics: for entity state events (account balance changed), use log compaction. Kafka keeps only the latest value per key, which naturally handles out-of-order and duplicate events.
  • Event sourcing: store every event as an append-only log. Rebuild entity state by replaying events. Deduplication is built into the event store.
  • Dead letter queue: if an event fails processing after all retries, send it to a DLQ topic for manual investigation. Do not block the consumer.
JAVA
package com.bank.events;

public class SeniorDedupKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal event dedup checklist:");
        System.out.println("  1. Use processed_events table with UNIQUE constraint on event_id");
        System.out.println("  2. Process + dedup INSERT in the SAME @Transactional method");
        System.out.println("  3. Use entity ID as Kafka message key for partition ordering");
        System.out.println("  4. Manual offset commit (enable.auto.commit=false)");
        System.out.println("  5. Add version field to events for out-of-order detection");
        System.out.println("  6. Set TTL on dedup table (match Kafka retention period)");
        System.out.println("  7. Dead letter queue for events that fail after all retries");
        System.out.println("  8. Monitor consumer lag per partition");
        System.out.println("  9. Consider log compaction for entity state events");
        System.out.println("  10. Idempotent producer (enable.idempotence=true) to prevent producer-side dupes");
    }
}
TEXT
Expected output:
Senior/Principal event dedup checklist:
  1. Use processed_events table with UNIQUE constraint on event_id
  2. Process + dedup INSERT in the SAME @Transactional method
  3. Use entity ID as Kafka message key for partition ordering
  4. Manual offset commit (enable.auto.commit=false)
  5. Add version field to events for out-of-order detection
  6. Set TTL on dedup table (match Kafka retention period)
  7. Dead letter queue for events that fail after all retries
  8. Monitor consumer lag per partition
  9. Consider log compaction for entity state events
  10. Idempotent producer (enable.idempotence=true) to prevent producer-side dupes

What are common interview follow-up questions?

  • How does Kafka's idempotent producer work? (Set enable.idempotence=true. The producer assigns a sequence number to each message. The broker rejects duplicates from the same producer session. This prevents producer-side duplicates but not consumer-side duplicates.)
  • What is the difference between deduplication and idempotency? (Deduplication detects and discards duplicate messages. Idempotency ensures that processing the same message multiple times has the same effect as processing it once. In practice, a dedup table provides idempotency.)
  • Can you achieve exactly-once processing from Kafka to a database? (Not natively. Kafka transactions work only for Kafka-to-Kafka. For Kafka-to-database, use at-least-once delivery with an idempotent consumer (dedup table + same-transaction pattern).)
  • How do you handle events that arrive weeks later due to a stuck consumer? (Set a reasonable TTL on the dedup table and a maximum age for events. If an event is older than the TTL, it might be re-processed as new. Design the business logic to handle this gracefully.)
  • What happens if the dedup table query is slow? (Use a database index on event_id. Consider a local cache or Bloom filter as a fast negative check before hitting the database.)

Visual Overview

DIAGRAM
 Idempotent Consumer Flow
 =========================

 Kafka Topic                  payment-service               PostgreSQL
 (order-events)               (consumer)                    (database)
      │                            │                              │
      │  Event: EVT-001            │                              │
      │  orderId: ORD-501          │                              │
      │───────────────────────────>│                              │
      │                            │  SELECT FROM processed_events│
      │                            │  WHERE event_id = 'EVT-001'  │
      │                            │─────────────────────────────>│
      │                            │  Result: NOT FOUND           │
      │                            │<─────────────────────────────│
      │                            │                              │
      │                            │  BEGIN TRANSACTION           │
      │                            │  INSERT payment_record       │
      │                            │  INSERT processed_event      │
      │                            │  COMMIT                      │
      │                            │─────────────────────────────>│
      │                            │  OK                          │
      │                            │<─────────────────────────────│
      │                            │                              │
      │  Event: EVT-001 (redelivered)                             │
      │───────────────────────────>│                              │
      │                            │  SELECT FROM processed_events│
      │                            │  WHERE event_id = 'EVT-001'  │
      │                            │─────────────────────────────>│
      │                            │  Result: FOUND               │
      │                            │<─────────────────────────────│
      │                            │                              │
      │                            │  SKIP (duplicate)            │
      │                            │                              │

Key Takeaway

Duplicate messages and out-of-order events are inherent properties of distributed event-driven systems. Kafka guarantees at-least-once delivery by default, which means a consumer might receive the same message more than once after a crash or rebalance. Events can also arrive out of order when a topic has multiple partitions or when producer retries reorder messages.

The standard solution for duplicates is an idempotent consumer backed by a processed_events table with a unique constraint on the event ID. Before processing any event, the consumer checks if the event ID exists in the table. If it does, the consumer skips the message. If it does not, the consumer processes the event and records the event ID in the same database transaction. This atomicity is critical: if the process succeeds but the recording fails, the event will be processed again on redelivery.

For out-of-order events, use the entity ID (like orderId) as the Kafka message key so that all events for the same entity land on the same partition, where Kafka guarantees ordering. For additional safety, include a version number in each event and compare it with the last processed version before processing.

In our National Bank of India project, order-service published OrderCreated events with the orderId as the Kafka message key. The payment-service consumed these events with a @KafkaListener and used a processed_events table to deduplicate. When Kafka redelivered an event after a consumer restart, the payment-service detected the duplicate event ID and skipped it, preventing Priya from being charged twice for the same order.

Remember one sentence: use a processed_events table with a unique constraint on event_id, and always process the event and record the event ID in the same database transaction.

---

Share
Use ← → arrow keys to navigate