Definition
- Manual acknowledgment – The consumer acknowledges a message only after the business work has durably committed, never on receipt.
- Idempotency – Processing the same message multiple times produces the same result, typically enforced with an inbox table of processed message IDs.
- Dead-letter queue (DLQ) – A holding queue for messages that always fail, so poison messages are parked for inspection instead of blocking the partition.
- Transactional outbox – The producer writes the event into an outbox table in the same transaction as the business change, eliminating the dual-write problem.
- Saga – For multi-step workflows, compensating transactions undo committed steps when a later step fails.
At National Bank of India, a back-office clerk takes a transfer slip from the inbox tray and starts working. If the clerk faints halfway, the rule is that the slip stays in the inbox because the clerk only moves it to the done tray after the work is fully finished. When the clerk recovers, the slip gets redone. To make sure Rajesh is not charged twice, each slip has a unique stamp the clerk checks first. Slips that keep causing problems are moved to a special tray for a supervisor.
Understanding the Concepts
What exactly happens when a consumer crashes?
With at-least-once delivery and manual acknowledgment, the consumer receives a message, processes it, and only then acknowledges. If it crashes before acknowledging, the broker redelivers the message.
package com.bank.messaging;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Component;
@Component
public class TransferEventConsumer {
private final TransferProcessor processor;
public TransferEventConsumer(TransferProcessor processor) {
this.processor = processor;
}
@KafkaListener(topics = "transfer-events", groupId = "credit-group")
public void onMessage(String payload, Acknowledgment ack) {
// 1. Process the business logic
processor.creditRecipient(payload);
// 2. Acknowledge ONLY after commit
ack.acknowledge();
System.out.println("Message acknowledged after successful processing");
// If crash happens between 1 and 2:
// -> Kafka sees no ack -> redelivers the message
// -> Message is processed AGAIN (must be idempotent!)
}
}
Crash scenarios:
Crash BEFORE processing: message redelivered, no harm
Crash DURING processing: partial work + redelivery = DUPLICATE risk
Crash AFTER processing, BEFORE ack: work done + redelivery = DUPLICATE
-> Idempotency handles both duplicate scenarios
How does idempotency make redelivery safe?
Before processing, check whether this message ID has already been handled. If yes, skip and acknowledge.
package com.bank.messaging;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class IdempotentCreditService {
private final ProcessedMessageRepository processedRepo;
private final AccountRepository accountRepo;
public IdempotentCreditService(ProcessedMessageRepository processedRepo,
AccountRepository accountRepo) {
this.processedRepo = processedRepo;
this.accountRepo = accountRepo;
}
@Transactional
public void creditRecipient(String messageId, String accountId, BigDecimal amount) {
// Idempotency check: has this message been processed?
if (processedRepo.existsById(messageId)) {
System.out.println("Message " + messageId + " already processed, skipping");
return;
}
// Process the credit
Account account = accountRepo.findById(accountId).orElseThrow();
account.credit(amount);
accountRepo.save(account);
// Record that this message was processed (in same transaction)
processedRepo.save(new ProcessedMessage(messageId));
System.out.println("Credited " + amount + " to " + accountId);
}
}
Expected output (first delivery):
Credited 50000.00 to SAV2002
Expected output (redelivery after crash):
Message MSG-001 already processed, skipping
How does the dead-letter queue handle poison messages?
Some messages will never succeed. After bounded retries, route them to a DLQ for inspection.
package com.bank.messaging;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.util.backoff.FixedBackOff;
@Configuration
public class KafkaErrorConfig {
@Bean
public DefaultErrorHandler errorHandler(
KafkaTemplate<String, String> kafkaTemplate) {
// After 3 retries with 1-second backoff, send to DLQ
DeadLetterPublishingRecoverer recoverer =
new DeadLetterPublishingRecoverer(kafkaTemplate);
return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));
}
}
Poison message handling:
Attempt 1: NullPointerException (account_id is null)
Attempt 2: NullPointerException
Attempt 3: NullPointerException
-> Sent to transfer-events.DLT (dead letter topic)
-> Normal messages continue processing
-> Ops team inspects DLT, fixes data, replays if needed
How does the transactional outbox protect the producer?
The dual-write problem: update DB then publish event. If the process crashes between them, the event is lost. The outbox pattern writes both in one transaction.
package com.bank.outbox;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class OutboxTransferService {
private final TransferRepository transferRepo;
private final OutboxRepository outboxRepo;
public OutboxTransferService(TransferRepository transferRepo,
OutboxRepository outboxRepo) {
this.transferRepo = transferRepo;
this.outboxRepo = outboxRepo;
}
@Transactional
public void executeTransfer(String from, String to, BigDecimal amount) {
// Both in ONE transaction: crash between them is impossible
transferRepo.save(new Transfer(from, to, amount));
outboxRepo.save(new OutboxEvent("TRANSFER_COMPLETED",
"{\"from\":\"" + from + "\",\"to\":\"" + to + "\"}"));
System.out.println("Transfer + outbox event committed atomically");
}
}
Expected output:
Transfer + outbox event committed atomically
Outbox relay publishes the event to Kafka separately.
Crash between DB write and Kafka publish is IMPOSSIBLE.
How does a circuit breaker protect the consumer?
If the consumer calls a downstream that is down, retrying every message wastes resources. A circuit breaker fails fast during a cooldown period.
package com.bank.messaging;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.stereotype.Service;
@Service
public class FraudCheckClient {
@CircuitBreaker(name = "fraudCheck", fallbackMethod = "fallback")
public String checkFraud(String accountId) {
return externalFraudService.check(accountId);
}
public String fallback(String accountId, Throwable t) {
System.out.println("Fraud service down, marking for manual review");
return "MANUAL_REVIEW";
}
}
Without circuit breaker:
Fraud service down -> every message retries fraud call 3x
-> 10,000 messages * 3 retries = 30,000 wasted calls
With circuit breaker:
Fraud service down -> circuit opens after 10 failures
-> Remaining messages get fallback instantly
-> Fraud service recovers, circuit half-opens, tests, closes
How does a Saga handle multi-step failures?
When a crash happens partway through a sequence of local transactions across services, a Saga runs compensating transactions to undo the committed steps.
package com.bank.saga;
public class SagaCompensation {
public static void main(String[] args) {
System.out.println("Multi-step transfer Saga:");
System.out.println(" Step 1: Debit from account (committed)");
System.out.println(" Step 2: Credit to account (committed)");
System.out.println(" Step 3: Notify recipient (FAILED - crash)");
System.out.println();
System.out.println("Compensation:");
System.out.println(" C2: Reverse credit to account");
System.out.println(" C1: Reverse debit from account");
System.out.println(" Result: money back to original state");
}
}
Expected output:
Multi-step transfer Saga:
Step 1: Debit from account (committed)
Step 2: Credit to account (committed)
Step 3: Notify recipient (FAILED - crash)
Compensation:
C2: Reverse credit to account
C1: Reverse debit from account
Result: money back to original state
How do these mechanisms compose?
Each mechanism covers a different failure mode. Together they provide end-to-end reliability.
| Mechanism | What it guarantees |
|---|---|
| Transactional outbox | Event is never lost (producer side) |
| Manual ack + retry | Message is delivered and retried (broker side) |
| Idempotency | Redelivery is safe (consumer side) |
| Dead-letter queue | Poison messages do not block good ones |
| Circuit breaker | Failing dependency does not drain threads |
| Saga | Multi-step workflow stays consistent |
Composition:
Outbox -> guarantees event published
Kafka at-least-once -> guarantees delivery
Idempotency -> makes redelivery safe
DLQ -> contains poison
Circuit breaker -> protects dependencies
Saga -> keeps multi-step workflow consistent
What should a senior engineer know?
The processed-message table must be cleaned up periodically with a TTL. The DLQ is not a trash can but an inspection queue. The ack-mode in Spring Kafka is configured as MANUAL for at-least-once guarantees.
spring:
kafka:
consumer:
enable-auto-commit: false
listener:
ack-mode: MANUAL
Senior details:
- enable-auto-commit: false + ack-mode: MANUAL = at-least-once
- Processed message table: TTL 7 days, indexed by message_id
- DLQ: alert on volume, inspect daily, replay after fix
- Outbox: poll-based for simplicity, Debezium CDC for low latency
Visual Overview
Async Crash Recovery Flow:
┌──────────────┐ outbox ┌──────────┐ publish ┌─────────┐
│ Producer │─────────▶│ Outbox │─────────▶│ Kafka │
│ (same TX as │ atomic │ Table │ relay │ Topic │
│ business │ │ │ │ │
│ change) │ └──────────┘ └────┬────┘
└──────────────┘ │
┌───────────┤
│ │
┌─────▼─────┐ │ redelivery
│ Consumer │ │ on crash
│ │◀───┘
│ 1. check │
│ inbox │
│ 2. process │
│ 3. ack │
└─────┬─────┘
│ if poison
┌─────▼─────┐
│ DLQ │
│ (inspect) │
└───────────┘
Key Takeaway
When a consumer crashes during asynchronous processing, the broker redelivers the unacknowledged message, so no work is lost. But the same message may be delivered more than once, which is the central design challenge. The layered defense starts with manual acknowledgment so messages are only acked after the business change commits, then idempotency so processing twice produces the same result, then a dead-letter queue so poison messages that always fail are parked instead of blocking good ones.
On the producer side, the transactional outbox guarantees events are never lost by writing the event atomically with the business change. A circuit breaker stops the consumer from flooding a dependency that is already down, and for multi-step workflows a Saga runs compensating transactions. These mechanisms compose: the outbox guarantees publication, at-least-once plus manual ack guarantees delivery, idempotency makes redelivery safe, the DLQ contains poison, and the Saga keeps multi-step work consistent.
At National Bank of India, a consumer once credited Priya, crashed before acknowledging, and the redelivered message credited her a second time. Pankaj added an inbox-based idempotency check keyed by message ID. A malformed message with a null account ID failed every retry and starved the partition, so they added a DLQ after three attempts. And a producer that updated the ledger then crashed before publishing left a downstream blind, so they moved to a transactional outbox.
Remember one sentence: a crash means the unacknowledged message is redelivered, so design for safe redelivery with manual ack, idempotent processing, a DLQ for poison, a transactional outbox so events are never lost, and a Saga for multi-step consistency.
---