Definition
- Dead Letter Topic (DLT) – a separate Kafka topic that receives messages which could not be processed successfully after a configured number of retries, keeping them safe for human inspection or automated replay.
- Poison pill – a message that always causes a processing failure no matter how many times it is retried, such as a malformed JSON payload or an unrecognized event type.
- Retry topic – an intermediate Kafka topic created by frameworks like Spring Kafka where failed messages wait with increasing backoff delays before being retried.
@RetryableTopic– a Spring Kafka annotation that automatically creates retry topics and a DLT for a listener, with configurable attempts, backoff, and DLT suffix.DeadLetterPublishingRecoverer– a Spring Kafka component that publishes a failedConsumerRecordto a DLT when the error handler gives up retrying.DefaultErrorHandler– the default Spring Kafka error handler that controls retry count, backoff, and the recoverer to invoke after exhaustion.
Think of National Bank of India processing incoming cheques. A clerk picks up a cheque (message) and tries to process it. If the cheque bounces because the issuing bank is temporarily unreachable, the clerk puts it in a retry tray and tries again after a short wait. If after three attempts the cheque still cannot be cleared, the clerk places it in a special locked drawer labeled "Unprocessable Cheques" (the Dead Letter Topic). A senior manager reviews that drawer every morning, investigates each cheque, and either fixes the issue and resubmits it or marks it as permanently rejected. The drawer ensures no cheque is silently lost.
Understanding the Concepts
What is a Dead Letter Topic (DLT)?
A Dead Letter Topic is a dedicated Kafka topic that acts as a final destination for messages that could not be processed after all configured retries have been exhausted. Instead of dropping the message or blocking the consumer, the framework publishes the failed record to the DLT, where it can be inspected, analyzed, and optionally replayed once the root cause is fixed.
In a banking system, the DLT protects you from losing critical financial events. If a transfer event from account SAV2001 cannot be processed because a downstream ledger service is unavailable, the event moves to the DLT after retries. No money is lost, and the operations team is alerted.
package com.bank.messaging;
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.TopicSuffixingStrategy;
import org.springframework.retry.annotation.Backoff;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.stereotype.Component;
@Component
public class TransferEventListener {
@RetryableTopic(
attempts = "4",
backoff = @Backoff(delay = 1000, multiplier = 2),
topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE,
dltTopicSuffix = "-dlt"
)
@KafkaListener(topics = "transfer-events", groupId = "settlement-group")
public void processTransfer(ConsumerRecord<String, String> record) {
// Process the transfer event
System.out.println("Processing transfer for account: " + record.key());
// If this throws, Spring retries with backoff, then sends to DLT
}
@DltHandler
public void handleDlt(ConsumerRecord<String, String> record) {
System.out.println("DLT received failed transfer: key=" + record.key()
+ " value=" + record.value());
// Log to database, alert operations team
}
}
Expected output (normal flow):
Processing transfer for account: SAV2001
Expected output (failure after 4 attempts):
Processing transfer for account: SAV2001 (attempt 1 - fails)
Processing transfer for account: SAV2001 (attempt 2 after 1s - fails)
Processing transfer for account: SAV2001 (attempt 3 after 2s - fails)
Processing transfer for account: SAV2001 (attempt 4 after 4s - fails)
DLT received failed transfer: key=SAV2001 value={...}
Why do we need a Dead Letter Topic?
Without a DLT, a failing message has only two outcomes: it blocks the partition forever while the consumer retries indefinitely, or it is silently skipped and lost. Both are unacceptable for financial events. A DLT provides a third, safe path: park the message, unblock the partition, and let a human or automated process deal with it later.
Consider a scenario at National Bank of India where Rajesh transfers money from SAV2001 to Priya's account. If the ledger service is down and the consumer has no DLT, the consumer either retries forever (blocking all subsequent transfers on that partition) or skips the event (losing the transfer). With a DLT, the consumer retries three times with backoff, then publishes the failed event to transfer-events-dlt, and moves on to the next message. The operations team is alerted and can replay the event once the ledger is back.
package com.bank.service;
import java.math.BigDecimal;
public class TransferService {
private final LedgerClient ledgerClient;
public TransferService(LedgerClient ledgerClient) {
this.ledgerClient = ledgerClient;
}
public void processTransfer(String fromAccount, String toAccount,
BigDecimal amount) {
// This may throw if the ledger service is down
ledgerClient.debit(fromAccount, amount);
ledgerClient.credit(toAccount, amount);
System.out.println("Transfer completed: " + fromAccount
+ " -> " + toAccount + " amount=" + amount);
}
}
Expected output (success):
Transfer completed: SAV2001 -> SAV3002 amount=50000.00
Expected output (ledger down):
Exception: LedgerServiceUnavailableException
-> Message sent to transfer-events-retry-0
-> Message sent to transfer-events-retry-1
-> Message sent to transfer-events-retry-2
-> Message sent to transfer-events-dlt
How does the DLT flow work internally?
When Spring Kafka's @RetryableTopic is configured with attempts = "4", the framework creates intermediate retry topics and a final DLT topic. On the first failure, the message is published to transfer-events-retry-0. After the backoff delay, a listener on that retry topic picks it up and retries. If it fails again, it moves to transfer-events-retry-1 with a longer backoff. After all attempts are exhausted, it lands in transfer-events-dlt.
Internally, each retry topic has its own consumer that polls, waits for the backoff to expire, and then invokes the original listener method. The DLT topic has a separate consumer that invokes the @DltHandler method. The original partition's consumer is not blocked during this process because the failed message has already moved to a different topic.
package com.bank.config;
import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class KafkaTopicConfig {
// Spring @RetryableTopic auto-creates these, but you can pre-create:
// transfer-events (main topic)
// transfer-events-retry-0 (1st retry, 1s delay)
// transfer-events-retry-1 (2nd retry, 2s delay)
// transfer-events-retry-2 (3rd retry, 4s delay)
// transfer-events-dlt (dead letter topic)
@Bean
public NewTopic transferEventsTopic() {
return new NewTopic("transfer-events", 6, (short) 3);
}
@Bean
public NewTopic transferEventsDlt() {
return new NewTopic("transfer-events-dlt", 6, (short) 3);
}
}
Topics created:
transfer-events -> 6 partitions, replication-factor 3
transfer-events-retry-0 -> auto-created by @RetryableTopic
transfer-events-retry-1 -> auto-created by @RetryableTopic
transfer-events-retry-2 -> auto-created by @RetryableTopic
transfer-events-dlt -> 6 partitions, replication-factor 3
How do you implement DLT programmatically with DefaultErrorHandler?
The annotation-driven @RetryableTopic is convenient, but sometimes you need programmatic control. The DefaultErrorHandler with a DeadLetterPublishingRecoverer gives you that. You configure the error handler in a ConcurrentKafkaListenerContainerFactory and specify the recoverer that publishes to the DLT.
package com.bank.config;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.TopicPartition;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;
@Configuration
public class KafkaErrorHandlerConfig {
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String>
kafkaListenerContainerFactory(
ConsumerFactory<String, String> consumerFactory,
KafkaTemplate<String, String> kafkaTemplate) {
DeadLetterPublishingRecoverer recoverer =
new DeadLetterPublishingRecoverer(kafkaTemplate,
(ConsumerRecord<?, ?> record, Exception ex) ->
new TopicPartition(
record.topic() + "-dlt", record.partition()));
// 3 retries, 2 second interval, then DLT
DefaultErrorHandler errorHandler =
new DefaultErrorHandler(recoverer, new FixedBackOff(2000L, 3));
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
factory.setCommonErrorHandler(errorHandler);
return factory;
}
}
Expected behavior:
- Message fails processing
- DefaultErrorHandler retries 3 times with 2s interval
- After 3 retries, DeadLetterPublishingRecoverer publishes to <topic>-dlt
- Consumer moves to next record
How is a DLT different from a retry topic?
A retry topic is an intermediate holding area where messages wait with a backoff delay before being retried. A DLT is the final destination after all retries have failed. The retry topic is temporary and optimistic (the message may succeed on retry), while the DLT is permanent and pessimistic (the message needs human intervention).
| Feature | Retry Topic | Dead Letter Topic |
|---|---|---|
| Purpose | Temporary hold for delayed retry | Final destination for failed messages |
| Message lifetime | Short (backoff duration) | Long (until manual intervention) |
| Automatic processing | Yes, retried after backoff | No, requires manual review or replay |
| Topic naming |
| When created | Per retry attempt level | Once, as the final catch-all |
| Consumer action | Re-invoke original listener | Invoke @DltHandler or log for ops |
package com.bank.messaging;
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.retry.annotation.Backoff;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.stereotype.Component;
@Component
public class PaymentEventListener {
// Retry topics: payment-events-retry-0, payment-events-retry-1
// DLT: payment-events-dlt
@RetryableTopic(
attempts = "3",
backoff = @Backoff(delay = 1000, multiplier = 2)
)
@KafkaListener(topics = "payment-events", groupId = "payment-group")
public void processPayment(ConsumerRecord<String, String> record) {
System.out.println("Processing payment: " + record.key());
// Throws on failure -> retry topic -> DLT
}
@DltHandler
public void handleFailedPayment(ConsumerRecord<String, String> record) {
System.out.println("Payment failed permanently: " + record.key());
// Alert the operations team
}
}
Flow: payment-events -> payment-events-retry-0 (1s) -> payment-events-retry-1 (2s) -> payment-events-dlt
What are common error handling strategies in Kafka consumers?
There are several strategies for handling errors in Kafka consumers, and the right choice depends on the error type. Transient errors like network timeouts should be retried. Permanent errors like malformed messages should go to the DLT immediately. Business validation errors may need custom routing.
The three main strategies are: retry with backoff (for transient errors), skip and log (for non-critical errors), and DLT (for errors that need investigation). You can combine these by classifying exceptions.
package com.bank.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.ExponentialBackOff;
import com.fasterxml.jackson.core.JsonParseException;
@Configuration
public class ClassifiedErrorHandlerConfig {
@Bean
public DefaultErrorHandler classifiedErrorHandler(
KafkaTemplate<String, String> kafkaTemplate) {
DeadLetterPublishingRecoverer recoverer =
new DeadLetterPublishingRecoverer(kafkaTemplate);
ExponentialBackOff backOff = new ExponentialBackOff(1000L, 2.0);
backOff.setMaxElapsedTime(10000L);
DefaultErrorHandler handler =
new DefaultErrorHandler(recoverer, backOff);
// Do not retry deserialization or JSON parse errors
// Send them to DLT immediately
handler.addNotRetryableExceptions(
JsonParseException.class,
IllegalArgumentException.class
);
return handler;
}
}
Behavior:
- JsonParseException -> sent to DLT immediately (no retry)
- IllegalArgumentException -> sent to DLT immediately (no retry)
- RuntimeException -> retried with exponential backoff, then DLT
- TimeoutException -> retried with exponential backoff, then DLT
What are common mistakes with DLT configuration?
The most common mistake is not having a DLT at all, which means the consumer either blocks or silently loses messages. Another mistake is retrying non-retryable exceptions, which wastes time and delays processing of valid messages.
Wrong approach: retrying a deserialization error that will never succeed.
package com.bank.messaging;
// WRONG: Retries a malformed JSON message 10 times before giving up.
// The message will never succeed because the JSON is broken.
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.retry.annotation.Backoff;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.stereotype.Component;
@Component
public class WrongRetryListener {
@RetryableTopic(
attempts = "10",
backoff = @Backoff(delay = 5000)
)
@KafkaListener(topics = "transfer-events", groupId = "wrong-group")
public void processTransfer(ConsumerRecord<String, String> record) {
// Malformed JSON causes JsonParseException on EVERY attempt
// 10 retries * 5s = 50 seconds wasted per bad message
TransferEvent event = parseJson(record.value());
System.out.println("Processed: " + event);
}
private TransferEvent parseJson(String json) {
throw new RuntimeException("Malformed JSON: " + json);
}
}
Correct approach: classify exceptions so non-retryable errors go to DLT immediately.
package com.bank.messaging;
// CORRECT: Non-retryable exceptions skip retry and go straight to DLT.
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.DltStrategy;
import org.springframework.retry.annotation.Backoff;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.stereotype.Component;
@Component
public class CorrectRetryListener {
@RetryableTopic(
attempts = "3",
backoff = @Backoff(delay = 1000, multiplier = 2),
exclude = { com.fasterxml.jackson.core.JsonParseException.class },
dltStrategy = DltStrategy.FAIL_ON_ERROR
)
@KafkaListener(topics = "transfer-events", groupId = "correct-group")
public void processTransfer(ConsumerRecord<String, String> record) {
TransferEvent event = parseJson(record.value());
System.out.println("Processed: " + event);
}
@DltHandler
public void handleDlt(ConsumerRecord<String, String> record) {
System.out.println("DLT: " + record.key() + " -> " + record.value());
}
private TransferEvent parseJson(String json) {
// Real parsing logic
return new TransferEvent();
}
}
WRONG behavior:
- Malformed message retried 10 times (50 seconds wasted)
- Partition blocked during retries
CORRECT behavior:
- Malformed message sent to DLT immediately (0 seconds wasted)
- Transient errors retried 3 times with backoff (1s, 2s, 4s)
What are the edge cases with Dead Letter Topics?
One edge case is when the DLT itself is unavailable. If the DLT topic does not exist or the producer cannot reach it, the DeadLetterPublishingRecoverer fails, and the original message is lost or the consumer blocks depending on your error handler configuration.
Another edge case is key and partition handling. By default, the DLT record keeps the same key as the original record, so ordering per key is preserved in the DLT. However, if the DLT has a different number of partitions, the key-to-partition mapping may differ.
package com.bank.service;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.TopicPartition;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
public class SafeDltRecoverer {
public static DeadLetterPublishingRecoverer create(
KafkaTemplate<String, String> kafkaTemplate) {
return new DeadLetterPublishingRecoverer(kafkaTemplate,
(ConsumerRecord<?, ?> record, Exception ex) -> {
// Preserve original partition if DLT has same partition count
String dltTopic = record.topic() + "-dlt";
System.out.println("Sending to DLT: topic=" + dltTopic
+ " key=" + record.key()
+ " partition=" + record.partition()
+ " error=" + ex.getMessage());
return new TopicPartition(dltTopic, record.partition());
});
}
}
Edge case output:
Sending to DLT: topic=transfer-events-dlt key=SAV2001 partition=2 error=Ledger unavailable
Edge case: If DLT has fewer partitions than source topic,
partition=5 would cause an error if DLT only has 3 partitions.
Solution: Match DLT partition count to source topic.
When should you use a Dead Letter Topic?
You should use a DLT whenever message loss is unacceptable and some messages may fail permanently. In banking, every financial event is critical, so a DLT is always present. The rule is simple: if the data has monetary or compliance value, use a DLT.
You should not use a DLT for high-volume, low-value telemetry where occasional loss is acceptable and DLT volume would be overwhelming.
package com.bank.messaging;
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.retry.annotation.Backoff;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.stereotype.Component;
@Component
public class AuditEventListener {
// Financial audit events: ALWAYS use DLT
@RetryableTopic(
attempts = "3",
backoff = @Backoff(delay = 2000, multiplier = 2)
)
@KafkaListener(topics = "audit-events", groupId = "audit-group")
public void processAuditEvent(ConsumerRecord<String, String> record) {
System.out.println("Audit event for account: " + record.key());
// Compliance requirement: every audit event must be processed
}
@DltHandler
public void handleAuditDlt(ConsumerRecord<String, String> record) {
System.out.println("CRITICAL: Audit event in DLT: " + record.key());
// Page on-call engineer immediately
}
}
Use DLT:
- Transfer events (monetary value)
- Audit events (compliance requirement)
- Loan application events (customer impact)
Skip DLT:
- Dashboard metrics (regenerated every minute)
- Click-stream analytics (high volume, low individual value)
What about monitoring and replaying DLT messages?
In production banking systems, a DLT without monitoring is useless. You need alerts when messages arrive in the DLT, dashboards showing DLT depth per topic, and a replay mechanism to reprocess messages once the root cause is fixed.
The replay pattern reads from the DLT and republishes to the original topic. Pankaj, the architect at National Bank of India, built a simple replay tool that reads DLT records, optionally transforms them, and sends them back to the source topic.
package com.bank.ops;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
@Component
public class DltReplayService {
private final KafkaTemplate<String, String> kafkaTemplate;
private volatile boolean replayEnabled = false;
public DltReplayService(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void enableReplay() {
this.replayEnabled = true;
System.out.println("DLT replay enabled");
}
@KafkaListener(topics = "transfer-events-dlt", groupId = "dlt-replay-group")
public void replayDltMessage(ConsumerRecord<String, String> record) {
if (replayEnabled) {
kafkaTemplate.send("transfer-events", record.key(), record.value());
System.out.println("Replayed DLT message: key=" + record.key());
} else {
System.out.println("DLT message parked: key=" + record.key());
}
}
}
Expected output (replay disabled):
DLT message parked: key=SAV2001
Expected output (replay enabled):
DLT replay enabled
Replayed DLT message: key=SAV2001
-> Message reprocessed by transfer-events consumer
What should a senior engineer know about DLT?
A senior engineer should understand that DLT is not a fire-and-forget dumping ground. Every message in the DLT represents a failed business operation that needs resolution. The DLT should have the same replication factor and partition count as the source topic. DLT messages should carry headers with the original topic, partition, offset, exception class, and exception message for debugging.
Spring Kafka automatically adds these headers: kafka_dlt-original-topic, kafka_dlt-original-partition, kafka_dlt-original-offset, kafka_dlt-exception-fqcn, and kafka_dlt-exception-message. A senior engineer monitors DLT consumer lag as a production health metric.
package com.bank.ops;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.header.Header;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import java.nio.charset.StandardCharsets;
@Component
public class DltInspector {
@KafkaListener(topics = "transfer-events-dlt", groupId = "dlt-inspector")
public void inspectDltRecord(ConsumerRecord<String, String> record) {
System.out.println("DLT Record:");
System.out.println(" Key: " + record.key());
System.out.println(" Value: " + record.value());
for (Header header : record.headers()) {
String value = new String(header.value(), StandardCharsets.UTF_8);
System.out.println(" Header: " + header.key() + " = " + value);
}
}
}
Expected output:
DLT Record:
Key: SAV2001
Value: {"transferId":"TRF-001","amount":50000.00}
Header: kafka_dlt-original-topic = transfer-events
Header: kafka_dlt-original-partition = 2
Header: kafka_dlt-original-offset = 1547
Header: kafka_dlt-exception-fqcn = com.bank.exception.LedgerUnavailableException
Header: kafka_dlt-exception-message = Ledger service timeout after 5000ms
Visual Overview
Dead Letter Topic Flow
=======================
Producer Kafka Cluster
(PaymentService) (transfer-events topic)
│ │
│ send(key=SAV2001, event) │
├─────────────────────────────▶│
│ │
│ ▼
┌──────────────┐
│ transfer- │
│ events │
│ (partition 2)│
└──────┬───────┘
│
▼
┌──────────────┐
│ Consumer │
│ (settlement- │
│ group) │
└──────┬───────┘
│
Process fails?
┌──────┴───────┐
│ │
No Yes
│ │
▼ ▼
Done ┌────────────────┐
│ transfer- │
│ events-retry-0 │
│ (backoff: 1s) │
└───────┬────────┘
│
Still fails?
┌──────┴───────┐
│ │
No Yes
│ │
▼ ▼
Done ┌────────────────┐
│ transfer- │
│ events-retry-1 │
│ (backoff: 2s) │
└───────┬────────┘
│
Still fails?
┌──────┴───────┐
│ │
No Yes
│ │
▼ ▼
Done ┌────────────────┐
│ transfer- │
│ events-dlt │
│ (Dead Letter) │
└───────┬────────┘
│
▼
┌────────────────┐
│ @DltHandler │
│ Log + Alert │
│ Operations │
└────────────────┘
Key Takeaway
A Dead Letter Topic is one of the most important patterns in any Kafka-based banking system. When a consumer cannot process a message after all retries, the DLT catches it instead of letting it be lost or blocking the partition. In National Bank of India's transfer processing pipeline, every failed transfer event moves through retry topics with exponential backoff and finally lands in the DLT, where the operations team can inspect it, fix the root cause, and replay it. Without a DLT, a poison pill message would either block the partition forever or be silently dropped, both of which are unacceptable for financial data.
Spring Kafka provides two approaches for DLT: the annotation-driven @RetryableTopic with @DltHandler for simple cases, and the programmatic DefaultErrorHandler with DeadLetterPublishingRecoverer for fine-grained control. The key design decision is classifying exceptions: transient errors like network timeouts should be retried, while permanent errors like malformed JSON should skip retry and go directly to the DLT. This classification prevents wasting time retrying messages that will never succeed.
In production, the DLT is not complete without monitoring and replay. Every DLT message carries headers with the original topic, partition, offset, and exception details. The operations team monitors DLT depth as a critical health metric and uses a replay service to resubmit messages once the root cause is fixed. The DLT itself should have the same replication factor as the source topic so failed messages are not lost to broker failures.
Remember one sentence: a Dead Letter Topic is the safety net that catches failed messages after all retries, ensuring no financial event is ever silently lost.
---