Definition
- Poison-pill message -- a record that a consumer can never successfully process, causing every retry to fail and blocking the partition indefinitely.
- Dead letter topic (DLT) -- a separate topic where failed records are published with error metadata so they can be inspected, triaged, and replayed offline.
- Bounded retries -- a configured maximum number of retry attempts before the record is routed to the DLT; prevents infinite loops.
- DefaultErrorHandler -- a Spring Kafka error handler that catches processing exceptions and routes unrecoverable records to a DLT.
- DeadLetterPublishingRecoverer -- a Spring Kafka recoverer that publishes the original record and error headers to a dead letter topic.
- Exception classification -- separating transient failures (worth retrying) from permanent failures (straight to DLT).
Imagine National Bank of India's settlement clerk processing a stack of transfer slips one by one. One slip is garbage the clerk cannot read, and the rule says do not move to the next slip until this one is done. The clerk is stuck forever while the whole stack waits. A dead letter topic is like a labeled problem tray: after a few honest tries, the clerk drops the bad slip into the problem tray, notes why, and moves on to the rest. A supervisor later inspects the problem tray and decides what to do.
Understanding the Concepts
What is a poison-pill message?
A poison pill is a record that a consumer can never process successfully. Every attempt fails with the same exception. Because Kafka processes records in offset order and only commits the offset after success, the consumer never advances past the poison pill, refetches it on every poll, and fails again forever. The entire partition is blocked.
package com.bank.error;
public class PoisonPillScenario {
public static void main(String[] args) {
System.out.println("Partition 3 records:");
System.out.println(" offset 100: valid transfer SAV2001");
System.out.println(" offset 101: POISON PILL (invalid amount: -50000)");
System.out.println(" offset 102: valid transfer SAV3045");
System.out.println(" offset 103: valid transfer SAV1122");
System.out.println();
System.out.println("Without DLT:");
System.out.println(" Consumer processes offset 100 -- OK");
System.out.println(" Consumer processes offset 101 -- FAIL");
System.out.println(" Consumer retries offset 101 -- FAIL");
System.out.println(" Consumer retries offset 101 -- FAIL (forever)");
System.out.println(" Offsets 102, 103 NEVER processed!");
}
}
Partition 3 records:
offset 100: valid transfer SAV2001
offset 101: POISON PILL (invalid amount: -50000)
offset 102: valid transfer SAV3045
offset 103: valid transfer SAV1122
Without DLT:
Consumer processes offset 100 -- OK
Consumer processes offset 101 -- FAIL
Consumer retries offset 101 -- FAIL
Consumer retries offset 101 -- FAIL (forever)
Offsets 102, 103 NEVER processed!
How does a dead letter topic fix this?
After a bounded number of retries, the consumer publishes the failing record to a dead letter topic with error metadata, commits the offset to move past it, and continues processing the rest of the partition.
package com.bank.config;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.util.backoff.FixedBackOff;
public class DltConfiguration {
public static DefaultErrorHandler errorHandler(
KafkaTemplate<String, String> kafkaTemplate) {
// After 3 retries with 1-second wait, publish to DLT
DeadLetterPublishingRecoverer recoverer =
new DeadLetterPublishingRecoverer(kafkaTemplate);
DefaultErrorHandler handler = new DefaultErrorHandler(
recoverer, new FixedBackOff(1000L, 3));
return handler;
}
public static void main(String[] args) {
System.out.println("DLT flow:");
System.out.println(" offset 101: try 1 -- FAIL");
System.out.println(" offset 101: try 2 -- FAIL (wait 1s)");
System.out.println(" offset 101: try 3 -- FAIL (wait 1s)");
System.out.println(" offset 101: publish to transfers.DLT");
System.out.println(" offset 101: commit, advance to 102");
System.out.println(" offset 102: process -- OK");
System.out.println(" Partition keeps flowing!");
}
}
DLT flow:
offset 101: try 1 -- FAIL
offset 101: try 2 -- FAIL (wait 1s)
offset 101: try 3 -- FAIL (wait 1s)
offset 101: publish to transfers.DLT
offset 101: commit, advance to 102
offset 102: process -- OK
Partition keeps flowing!
How do you classify exceptions?
Not all failures deserve retries. Transient failures like a database timeout may succeed on retry. Permanent failures like a validation error will never succeed. Route permanent failures straight to the DLT without wasting retries.
package com.bank.config;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.util.backoff.FixedBackOff;
public class ExceptionClassification {
public static DefaultErrorHandler classifiedHandler(
DeadLetterPublishingRecoverer recoverer) {
DefaultErrorHandler handler = new DefaultErrorHandler(
recoverer, new FixedBackOff(1000L, 3));
// These go straight to DLT -- no retries
handler.addNotRetryableExceptions(
IllegalArgumentException.class, // validation error
com.fasterxml.jackson.core.JsonParseException.class // bad JSON
);
return handler;
}
public static void main(String[] args) {
System.out.println("Exception classification:");
System.out.println(" Transient (retry): DatabaseTimeoutException");
System.out.println(" Transient (retry): SocketTimeoutException");
System.out.println(" Permanent (DLT): IllegalArgumentException");
System.out.println(" Permanent (DLT): JsonParseException");
System.out.println(" Permanent (DLT): SerializationException");
}
}
Exception classification:
Transient (retry): DatabaseTimeoutException
Transient (retry): SocketTimeoutException
Permanent (DLT): IllegalArgumentException
Permanent (DLT): JsonParseException
Permanent (DLT): SerializationException
How do you handle deserialization poison pills?
Deserialization failures happen inside the Kafka client before your business logic runs. A safe deserializer wraps the real deserializer and produces a null value with the failure in a header instead of crashing.
package com.bank.config;
import java.util.Properties;
public class SafeDeserializerConfig {
public static Properties config() {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "settlement-service");
// Wrap the real deserializer
props.put("value.deserializer",
"org.springframework.kafka.support.serializer.ErrorHandlingDeserializer");
props.put("spring.deserializer.value.delegate.class",
"org.springframework.kafka.support.serializer.JsonDeserializer");
return props;
}
public static void main(String[] args) {
System.out.println("Without ErrorHandlingDeserializer:");
System.out.println(" Bad bytes -> SerializationException -> consumer stuck");
System.out.println();
System.out.println("With ErrorHandlingDeserializer:");
System.out.println(" Bad bytes -> null value + error header -> DLT");
}
}
Without ErrorHandlingDeserializer:
Bad bytes -> SerializationException -> consumer stuck
With ErrorHandlingDeserializer:
Bad bytes -> null value + error header -> DLT
What metadata should the DLT record carry?
The dead letter record must carry enough context to debug and replay.
package com.bank.dlt;
public class DltMetadata {
public static void main(String[] args) {
System.out.println("DLT record headers (auto-added by Spring):");
System.out.println(" kafka_dlt-original-topic : transfers");
System.out.println(" kafka_dlt-original-partition : 3");
System.out.println(" kafka_dlt-original-offset : 101");
System.out.println(" kafka_dlt-original-timestamp : 1719849600000");
System.out.println(" kafka_dlt-exception-fqcn : IllegalArgumentException");
System.out.println(" kafka_dlt-exception-message : Negative amount");
System.out.println(" kafka_dlt-exception-stacktrace: ...");
}
}
DLT record headers (auto-added by Spring):
kafka_dlt-original-topic : transfers
kafka_dlt-original-partition : 3
kafka_dlt-original-offset : 101
kafka_dlt-original-timestamp : 1719849600000
kafka_dlt-exception-fqcn : IllegalArgumentException
kafka_dlt-exception-message : Negative amount
kafka_dlt-exception-stacktrace: ...
What are common mistakes?
package com.bank.mistake;
public class PoisonPillMistakes {
public static void main(String[] args) {
// WRONG: infinite retries on a permanent failure
System.out.println("WRONG: retries=MAX on a validation error");
System.out.println(" Consumer retries forever, partition stuck");
System.out.println();
// CORRECT: classify and bound retries
System.out.println("CORRECT: 3 retries, then DLT, validation is not-retryable");
System.out.println();
// WRONG: DLT with no monitoring
System.out.println("WRONG: DLT exists but nobody watches it");
System.out.println(" Failures accumulate silently -- data loss");
System.out.println();
// CORRECT: alert on DLT depth, assign ownership
System.out.println("CORRECT: alert on transfers.DLT lag > 0");
}
}
WRONG: retries=MAX on a validation error
Consumer retries forever, partition stuck
CORRECT: 3 retries, then DLT, validation is not-retryable
WRONG: DLT exists but nobody watches it
Failures accumulate silently -- data loss
CORRECT: alert on transfers.DLT lag > 0
How does Spring Kafka handle poison pills declaratively?
Spring Kafka provides DefaultErrorHandler with DeadLetterPublishingRecoverer that handles the retry-then-DLT flow automatically. You configure the backoff policy and the recoverer, and any message that exhausts retries is published to the DLT topic with all the failure headers attached. The ErrorHandlingDeserializer wrapper catches deserialization failures before they reach your listener.
package com.bank.kafka;
public class SpringKafkaDltConfig {
public static void main(String[] args) {
System.out.println("Spring Kafka declarative DLT handling:");
System.out.println();
System.out.println(" // 1. Wrap deserializer for poison deserialization");
System.out.println(" props.put(VALUE_DESERIALIZER_CLASS_CONFIG,");
System.out.println(" ErrorHandlingDeserializer.class);");
System.out.println(" props.put(SPRING_DESERIALIZER_VALUE_DELEGATE_CLASS,");
System.out.println(" StringDeserializer.class);");
System.out.println();
System.out.println(" // 2. Configure error handler with DLT recoverer");
System.out.println(" DefaultErrorHandler handler = new DefaultErrorHandler(");
System.out.println(" new DeadLetterPublishingRecoverer(kafkaTemplate),");
System.out.println(" new FixedBackOff(1000L, 3L)");
System.out.println(" );");
System.out.println(" // 3. Non-retryable exceptions skip straight to DLT");
System.out.println(" handler.addNotRetryableExceptions(");
System.out.println(" DeserializationException.class,");
System.out.println(" ValidationException.class);");
}
}
Expected output:
Spring Kafka declarative DLT handling:
// 1. Wrap deserializer for poison deserialization
props.put(VALUE_DESERIALIZER_CLASS_CONFIG,
ErrorHandlingDeserializer.class);
props.put(SPRING_DESERIALIZER_VALUE_DELEGATE_CLASS,
StringDeserializer.class);
// 2. Configure error handler with DLT recoverer
DefaultErrorHandler handler = new DefaultErrorHandler(
new DeadLetterPublishingRecoverer(kafkaTemplate),
new FixedBackOff(1000L, 3L)
);
// 3. Non-retryable exceptions skip straight to DLT
handler.addNotRetryableExceptions(
DeserializationException.class,
ValidationException.class);
What should a senior engineer know?
A dead letter topic is not a fix -- it is an isolation mechanism. Without monitoring on its depth, root-cause classification, a replay procedure, and clear ownership, it quietly becomes a graveyard where failures pile up unnoticed. Consumers should be idempotent so that when you eventually replay a dead letter record, you do not create duplicate side effects.
Visual Overview
Poison Pill Handling with Dead Letter Topic
Partition 3
┌────────┬────────┬────────┬────────┐
│ off=100│ off=101│ off=102│ off=103│
│ valid │ POISON │ valid │ valid │
└───┬────┴───┬────┴───┬────┴───┬────┘
│ │ │ │
▼ ▼ │ │
Process Retry 1 │ │
OK FAIL │ │
Retry 2 │ │
FAIL │ │
Retry 3 │ │
FAIL │ │
│ │ │
▼ │ │
┌─────────────────┐ │ │
│ transfers.DLT │ │ │
│ off=101 + error │ │ │
│ metadata │ │ │
└─────────────────┘ │ │
│ │ │
commit past │ │
│ ▼ ▼
Process Process Process
OK OK OK
Key Takeaway
A poison pill is a record a consumer can never process, and the danger is that it blocks the entire partition. Kafka processes records in offset order and only commits after success, so one bad record means the consumer never advances, refetches it forever, and stalls every record behind it. In a bank, that means settlement for a whole partition grinds to a halt.
The standard fix is a dead letter topic. Wrap processing in bounded retries, and after the last attempt, publish the original record plus error metadata to a DLT and commit the offset. Classify exceptions so transient failures like database timeouts are retried, while permanent failures like validation errors skip straight to the DLT. For deserialization poison pills, use the ErrorHandlingDeserializer so the consumer does not crash before the error handler even runs.
At National Bank of India, a producer bug once emitted a transfer with a negative amount, and before the DLT was in place it stalled an entire settlement partition. Rajesh added bounded retries with a DLT, exception classification, and monitoring on DLT depth. Now a poison pill is quarantined, the partition keeps flowing, and the team gets paged to investigate.
Remember one sentence: a dead letter topic isolates a poison pill so the partition keeps flowing, but it must be monitored and owned, or it becomes a silent graveyard.
---