Definition
@RetryableTopic implements non-blocking retry and dead letter topic patterns declaratively in Spring Kafka.
- Non-blocking retry – failed records are forwarded to a dedicated retry topic instead of retried in place, so the main consumer continues immediately.
- Retry topic – a transient holding stage where the record is reprocessed after a delay; e.g.,
settlements-retry-0,settlements-retry-1. - Dead letter topic (DLT) – the terminal parking lot for records that exhausted all attempts; e.g.,
settlements-dlt. - Bounded attempts – a hard cap on the number of retry stages, guaranteeing termination.
- Exception classification – permanent failures like
ValidationExceptionskip retries and go straight to the DLT.
Picture National Bank of India processing a transfer that fails because a downstream service is briefly busy. Instead of the clerk standing at the counter retrying the same form over and over and blocking the whole line, @RetryableTopic moves that form to a sequence of waiting rooms: a one-second room, then a two-second room, then a four-second room. Each waiting room gives the downstream service more time to recover. If the form still fails after all waiting rooms, it goes to the permanent problem desk (the DLT) so a human can look at it.
Understanding the Concepts
What is the problem with in-place (blocking) retries?
The classic in-place retry done by DefaultErrorHandler is blocking: while it retries a record with backoff, the consumer thread for that partition is held, so every record behind it waits. For long backoffs or many retries this can stall the entire partition for minutes.
Blocking retry (DefaultErrorHandler):
Record A fails -> retry in 1s -> retry in 2s -> retry in 4s
During those 7 seconds, records B, C, D are WAITING
Partition throughput: zero for 7 seconds
How does @RetryableTopic make retries non-blocking?
@RetryableTopic moves the failed record off the main topic onto a dedicated retry topic and lets the main consumer continue immediately. A separate consumer reads the retry topic after the backoff delay.
package com.bank.kafka;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.DltStrategy;
import org.springframework.retry.annotation.Backoff;
import org.springframework.stereotype.Component;
@Component
public class NonBlockingRetryListener {
@RetryableTopic(
attempts = "4",
backoff = @Backoff(delay = 1000, multiplier = 2.0),
dltStrategy = DltStrategy.FAIL_ON_ERROR
)
@KafkaListener(topics = "settlements", groupId = "settlement-group")
public void onSettlement(String event) {
System.out.println("Processing: " + event);
// If this throws, record moves to retry topic, main consumer continues
settlementService.process(event);
}
@org.springframework.kafka.annotation.DltHandler
public void handleDlt(String event) {
System.out.println("DLT: " + event + " -- alerting ops team");
}
}
Processing: {"account":"SAV2001","amount":50000}
# If it fails:
# -> settlements-retry-0 (after 1s)
# -> settlements-retry-1 (after 2s)
# -> settlements-retry-2 (after 4s)
# -> settlements-dlt (terminal)
What topics does @RetryableTopic create?
Spring auto-creates retry topics and the DLT based on the original topic name and the number of attempts.
Original topic: settlements
Retry topic 0: settlements-retry-0 (delay: 1 second)
Retry topic 1: settlements-retry-1 (delay: 2 seconds)
Retry topic 2: settlements-retry-2 (delay: 4 seconds)
Dead letter topic: settlements-dlt (terminal)
# Each retry topic has its own consumer that waits the delay before reading
How is a retry topic different from the DLT?
A retry topic is a transient stage where records are expected to succeed on a later attempt once a transient problem clears. The DLT is the end of the line: records there have failed every attempt and will not be retried automatically.
| Aspect | Retry Topic | Dead Letter Topic |
|---|---|---|
| Purpose | Temporary holding for reprocessing | Terminal parking for failed records |
| Records reprocessed? | Yes, after delay | No (unless manual replay) |
| Expected outcome | Success after transient fix | Human inspection / alerting |
| Consumer | Automatic, reads after delay | Optional @DltHandler or separate consumer |
package com.bank.kafka;
public class RetryVsDltDemo {
public static void main(String[] args) {
System.out.println("Retry topic: record will be tried again after delay");
System.out.println("DLT: record will NOT be tried again automatically");
System.out.println("Retry topic: transient failure recovery");
System.out.println("DLT: permanent failure parking");
}
}
Retry topic: record will be tried again after delay
DLT: record will NOT be tried again automatically
Retry topic: transient failure recovery
DLT: permanent failure parking
How do you prevent infinite retry loops?
Two controls prevent infinite loops. First, bounded attempts: the attempts value is a hard cap. Second, exception classification: permanent failures skip retries entirely.
package com.bank.kafka;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.retry.annotation.Backoff;
import org.springframework.stereotype.Component;
@Component
public class ClassifiedRetryListener {
@RetryableTopic(
attempts = "4",
backoff = @Backoff(delay = 1000, multiplier = 2.0),
exclude = { javax.validation.ValidationException.class } // skip retries for validation errors
)
@KafkaListener(topics = "settlements", groupId = "settlement-group")
public void onSettlement(String event) {
validate(event); // ValidationException -> DLT immediately
process(event); // DB timeout -> retry through stages
}
void validate(String event) { /* ... */ }
void process(String event) { /* ... */ }
}
# Transient error (DB timeout): retry-0 -> retry-1 -> retry-2 -> DLT
# Permanent error (invalid account): skip retries -> DLT immediately
How does @DltHandler process dead letter records?
You can annotate a method with @DltHandler to process whatever lands in the DLT, for example to persist it for triage or send an alert.
package com.bank.kafka;
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.stereotype.Component;
@Component
public class DltProcessor {
@DltHandler
public void handleDlt(String event) {
System.out.println("DLT received: " + event);
// Save to a triage table for ops team
// Send Slack alert
// Log for audit trail
}
}
DLT received: {"account":"SAV2001","amount":50000}
What are the trade-offs of non-blocking retries?
Non-blocking retries change ordering because a retried record is reprocessed later than records that came after it on the main topic. Use this pattern only where strict per-key ordering is not required. Each retry topic needs partitions and consumers, adding operational overhead.
# Ordering impact:
# Main topic: [A, B, C] (A fails)
# A moves to retry topic
# B and C process normally on main topic
# A is reprocessed from retry topic AFTER B and C
# If A and B are for the same account and order matters: use blocking retries
# If order does not matter (e.g., independent events): use non-blocking retries
What should a senior engineer know?
Keep processing idempotent because a record can be delivered more than once across retry stages. Blocking retries via DefaultErrorHandler remain the better choice when strict per-key ordering must be preserved and recovery is expected to be fast. Each retry topic adds operational overhead (partitions, consumers, monitoring), so size the retry chain appropriately.
# Monitor retry topics and DLT
kafka-consumer-groups.sh --describe --group settlement-group \
--bootstrap-server kafka-1:9092
# Expected: retry topics should have low lag (records move through quickly)
# DLT should have near-zero lag (triaged regularly)
Visual Overview
@RetryableTopic Flow
Main topic: settlements
│
▼
Listener processes record
│
├── success ──→ offset committed ✓
│
└── failure ──→ forward to retry topic
│
▼
settlements-retry-0 (1s delay)
│
├── success ✓
└── failure ──→ settlements-retry-1 (2s delay)
│
├── success ✓
└── failure ──→ settlements-retry-2 (4s delay)
│
├── success ✓
└── failure ──→ settlements-dlt
│
▼
@DltHandler
(alert + triage)
Main consumer continues immediately after forwarding!
Records B, C, D are NOT blocked while A is in retry.
Key Takeaway
@RetryableTopic is Spring Kafka's declarative way to get non-blocking retries ending in a dead letter topic. Instead of holding the consumer thread while retrying, it forwards the failing record onto a dedicated retry topic and lets the main consumer continue immediately. Separate consumers read each retry topic after a delay, and after a bounded number of attempts the record lands in the DLT.
The distinction an interviewer wants is between a retry topic and the DLT. A retry topic is a transient holding stage where the record will be reprocessed after a delay in the hope that a temporary problem has cleared. The DLT is the end of the line, holding records that failed every attempt, kept for alerting, inspection, and possibly manual replay. You prevent infinite loops with bounded attempts and exception classification.
In the settlement service at National Bank of India, this fixed a real pain point where in-place retries were stalling the whole flow while Rajesh's fraud-scoring call was slow. With retry topics, a transient scoring failure was spaced out over one, two, and four seconds while the main flow kept moving, and it usually recovered by the second stage, whereas a transfer with a bad account was classified permanent and parked in the DLT instantly.
Remember one sentence: @RetryableTopic gives non-blocking retries through separate retry topics with growing backoff, ending in a DLT, and bounded attempts plus exception classification prevent infinite loops.
---