How do you stop one bad payload from blocking the partition, and what does DefaultErrorHandler do? | Kafka & Messaging

Q: How does ErrorHandlingDeserializer fix the poison pill?

You configure the value deserializer as ErrorHandlingDeserializer and tell it the real delegate (e.g., JsonDeserializer). When the delegate throws, the wrapper catches the exception and produces a record with a null value and the failure stored in headers. This converts a fatal pollthread exception

Definition

Wrap the deserializer so a bad payload becomes a handled error, and let DefaultErrorHandler retry with backoff, classify the exception, and route exhausted records to a dead letter topic.

Handling poison pills and deserialization failures requires two layers of protection in Spring Kafka.

Poison pill – a record that always fails to process; without handling, it retries forever and blocks the partition.
ErrorHandlingDeserializer – wraps the delegate deserializer; catches exceptions and converts them into recoverable errors with the failure stored in headers.
DefaultErrorHandler – retries failed records with a configurable BackOff, classifies exceptions as retryable vs non-retryable, and invokes a recoverer on exhaustion.
DeadLetterPublishingRecoverer – publishes exhausted records to a dead letter topic (.DLT suffix) with headers recording the original topic, partition, offset, and exception.
Exception classification – marking exceptions like DeserializationException as non-retryable so they skip retries and go straight to the DLT.

Imagine a mail clerk at National Bank of India who hits a form so garbled they cannot read it. Without protection, the clerk keeps trying to read the same unreadable form forever and the entire queue freezes. ErrorHandlingDeserializer is the rule that says if a form cannot be opened, set it aside as a flagged problem. DefaultErrorHandler is the supervisor who decides whether to try again a few times with a pause, recognizes which problems are hopeless versus temporary, and finally drops the truly broken forms into the problem tray (the DLT) so the real queue keeps flowing for Rajesh and Priya.

Understanding the Concepts

What is a poison pill in Kafka?

A poison pill is a record that always fails to process. Because Kafka will not advance the offset past an unhandled failure, the same record gets retried forever and blocks every good event behind it on that partition.

TEXT

Poison pill scenario:
1. Consumer reads offset 5: malformed JSON
2. Deserializer throws: JsonParseException
3. Exception crashes the poll thread
4. Next poll: reads offset 5 again (offset never advanced)
5. Deserializer throws again
6. Infinite loop -- partition stuck forever

How does ErrorHandlingDeserializer fix the poison pill?

You configure the value deserializer as ErrorHandlingDeserializer and tell it the real delegate (e.g., JsonDeserializer). When the delegate throws, the wrapper catches the exception and produces a record with a null value and the failure stored in headers. This converts a fatal poll-thread exception into a recoverable record.

JAVA

package com.bank.kafka;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.support.serializer.ErrorHandlingDeserializer;
import org.springframework.kafka.support.serializer.JsonDeserializer;
import java.util.Map;

@Configuration
public class SafeDeserializerConfig {

    @Bean
    public ConsumerFactory<String, Object> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(Map.of(
            ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092",
            ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class,
            ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class,
            ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class.getName(),
            JsonDeserializer.TRUSTED_PACKAGES, "com.bank.model",
            JsonDeserializer.VALUE_DEFAULT_TYPE, "com.bank.model.TxnEvent"
        ));
    }
}

TEXT

# Malformed JSON arrives: {"amount":INVALID}
# Without ErrorHandlingDeserializer: partition stuck in infinite loop
# With ErrorHandlingDeserializer: error flows to DefaultErrorHandler

What does DefaultErrorHandler do?

DefaultErrorHandler does three things: retries with configurable backoff, classifies exceptions, and invokes a recoverer when retries are exhausted.

JAVA

package com.bank.kafka;

import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.support.serializer.DeserializationException;
import org.springframework.util.backoff.FixedBackOff;

@Configuration
public class ErrorHandlerConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> template) {
        // Recoverer: send exhausted records to DLT
        DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(template);

        // Retry 3 times with 1-second backoff
        DefaultErrorHandler handler =
            new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3L));

        // Deserialization errors are NEVER retryable (bytes won't change)
        handler.addNotRetryableExceptions(DeserializationException.class);

        return handler;
    }
}

TEXT

# Transient DB error: retry 1 (1s) -> retry 2 (1s) -> retry 3 (1s) -> DLT
# Deserialization error: skip retries -> DLT immediately

How does DeadLetterPublishingRecoverer work?

It publishes the failed record to a dead letter topic (by convention the original topic name with .DLT suffix) with headers recording the original topic, partition, offset, exception class, and message.

TEXT

# Original record:
# Topic: settlements  Partition: 0  Offset: 42  Key: SAV2001

# After exhausting retries -> published to:
# Topic: settlements.DLT  Partition: 0
# Headers:
#   kafka_dlt-original-topic: settlements
#   kafka_dlt-original-partition: 0
#   kafka_dlt-original-offset: 42
#   kafka_dlt-exception-fqcn: org.springframework.kafka.support.serializer.DeserializationException
#   kafka_dlt-exception-message: Failed to deserialize value

Why must deserialization failures be non-retryable?

The bytes in a Kafka record will never change. Retrying a deserialization error wastes time and widens the replay window. Mark it as non-retryable so it goes straight to the DLT.

JAVA

package com.bank.kafka;

public class RetryClassificationDemo {
    public static void main(String[] args) {
        // Non-retryable: bytes won't change, retrying is pointless
        System.out.println("DeserializationException -> NOT retryable -> DLT immediately");
        System.out.println("ValidationException     -> NOT retryable -> DLT immediately");

        // Retryable: downstream may recover
        System.out.println("DatabaseTimeoutException -> retryable -> retry 3x -> then DLT");
        System.out.println("ServiceUnavailableException -> retryable -> retry 3x -> then DLT");
    }
}

TEXT

DeserializationException -> NOT retryable -> DLT immediately
ValidationException     -> NOT retryable -> DLT immediately
DatabaseTimeoutException -> retryable -> retry 3x -> then DLT
ServiceUnavailableException -> retryable -> retry 3x -> then DLT

What are common mistakes with error handling?

The most common mistake is not wrapping the deserializer, so a single malformed record blocks the entire partition. The second mistake is retrying non-retryable exceptions.

JAVA

package com.bank.kafka;

public class ErrorHandlingMistakes {
    public static void main(String[] args) {
        // WRONG: no ErrorHandlingDeserializer
        System.out.println("WRONG: raw JsonDeserializer -- one bad record blocks partition");

        // WRONG: retrying DeserializationException
        System.out.println("WRONG: retrying deserialization error 3 times -- bytes never change");

        // WRONG: DLT with no monitoring
        System.out.println("WRONG: DLT exists but nobody watches it -- events silently lost");

        // CORRECT: ErrorHandlingDeserializer + classify exceptions + monitor DLT
        System.out.println("CORRECT: wrap deserializer, classify errors, alert on DLT records");
    }
}

TEXT

WRONG: raw JsonDeserializer -- one bad record blocks partition
WRONG: retrying deserialization error 3 times -- bytes never change
WRONG: DLT exists but nobody watches it -- events silently lost
CORRECT: wrap deserializer, classify errors, alert on DLT records

How should you monitor and triage the DLT?

The DLT is not a dumping ground. It needs alerting and a triage process, otherwise it silently accumulates lost events. Set up a consumer or dashboard that monitors the DLT topic and triggers alerts.

BASH

# Monitor DLT lag (should be 0 if triaged regularly)
kafka-consumer-groups.sh --describe --group dlt-monitor-group \
  --bootstrap-server kafka-1:9092

# Alert rule: any record appearing in *.DLT topics triggers a Slack alert
# Triage process: inspect headers, fix upstream, replay if needed

What should a senior engineer know?

Because retries mean a record can reach the listener multiple times, the listener must stay idempotent. At-least-once semantics combined with retries make idempotency non-negotiable. The ErrorHandlingDeserializer is not optional in production. And the DLT must have its own retention, monitoring, and replay workflow.

JAVA

package com.bank.kafka;

import java.math.BigDecimal;

public class IdempotentWithRetries {
    public void settle(String txnId, String accountId, BigDecimal amount) {
        // INSERT INTO settlements (txn_id, account_id, amount)
        // ON CONFLICT (txn_id) DO NOTHING
        System.out.println("Idempotent settle: txn=" + txnId + " account=" + accountId);
        System.out.println("Duplicate insert is harmless no-op");
    }
}

TEXT

Idempotent settle: txn=TXN-12345 account=SAV2001
Duplicate insert is harmless no-op

Visual Overview

DIAGRAM

  Error Handling Pipeline

  Kafka Record
       │
       ▼
  ErrorHandlingDeserializer
       │
       ├── deserialization OK ──→ Listener method
       │                              │
       │                              ├── success ──→ offset committed ✓
       │                              │
       │                              └── exception ──→ DefaultErrorHandler
       │                                                    │
       │                                   ┌────────────────┤
       │                                   │                │
       │                              retryable?       non-retryable?
       │                                   │                │
       │                              retry with       skip retries
       │                              BackOff              │
       │                                   │                │
       │                              exhausted?            │
       │                                   │                │
       │                                   ▼                ▼
       │                          DeadLetterPublishingRecoverer
       │                                   │
       │                                   ▼
       │                           settlements.DLT topic
       │
       └── deserialization FAIL ──→ DefaultErrorHandler
                                   (non-retryable) ──→ DLT immediately

Key Takeaway

A poison pill is a record that always fails, and without protection it retries forever and blocks the entire partition. The fix has two layers. First, ErrorHandlingDeserializer wraps the real deserializer and catches exceptions, converting bad bytes into a handled error that flows to the error handler instead of crashing the poll loop. Second, DefaultErrorHandler retries transient failures with configurable backoff, classifies permanent failures as non-retryable so they skip retries, and routes exhausted records to a dead letter topic.

In the settlement service at National Bank of India, one corrupt transfer event from a buggy upstream was blocking a whole partition and stalling Rajesh's legitimate settlements. After wrapping the deserializer and configuring the handler, the corrupt event went straight to the DLT because deserialization errors are non-retryable, a transient database blip on another event was absorbed by a short retry with backoff, and the partition never stalled again.

The senior nuances are that deserialization failures must always be non-retryable because the bytes will never change, the DLT needs its own alerting and triage process or it silently fills with lost money events, and because retries mean a record can reach the listener multiple times, the listener must stay idempotent.

Remember one sentence: ErrorHandlingDeserializer prevents poison pills, DefaultErrorHandler classifies and retries, and DeadLetterPublishingRecoverer parks the hopeless records so the partition keeps moving.

---