Definition
- Tiered retry topics -- a non-blocking retry pattern where a failed record is republished to dedicated retry topics with increasing delays instead of blocking the main consumer.
- Retry ladder -- the graduating chain of topics: main -> retry.1s -> retry.5s -> retry.30s -> DLQ, with each tier adding a longer delay.
- Exponential backoff -- delays that grow geometrically (1s, 5s, 30s) to give a failing dependency time to recover.
- Jitter -- a random spread added to each delay to prevent a thundering-herd storm when many records fail at the same instant.
- Failure classification -- separating transient failures (worth retrying) from permanent failures (straight to DLQ without wasting tiers).
- @RetryableTopic -- a Spring Kafka annotation that auto-creates retry and DLT topics with configurable attempts, backoff multiplier, and exception exclusions.
Imagine National Bank of India's settlement desk hitting a temporary outage in the credit bureau. Instead of the clerk freezing on one form while the bureau is down, the form is dropped into a try again in 1 second tray, then a try again in 5 seconds tray, then a try again in 30 seconds tray, each handled by a different clerk who only picks up forms after the delay. If it still fails, it goes to the problem tray. Jitter is like telling clerks not to all retry at the exact same tick, so they do not stampede the bureau the moment it recovers.
Understanding the Concepts
Why not retry in place on the main consumer?
Blocking retries on the main consumer stall the whole partition while sleeping between attempts. One slow record delays everything behind it. Retry topics solve this by moving the waiting off the main consumer.
package com.bank.retry;
public class BlockingVsNonBlocking {
public static void main(String[] args) {
System.out.println("Blocking retry (BAD):");
System.out.println(" offset 100: process -> FAIL");
System.out.println(" sleep 1s, retry -> FAIL");
System.out.println(" sleep 5s, retry -> FAIL");
System.out.println(" Partition STALLED for 6+ seconds");
System.out.println(" offset 101, 102, 103 waiting...");
System.out.println();
System.out.println("Non-blocking retry topics (GOOD):");
System.out.println(" offset 100: process -> FAIL");
System.out.println(" Publish to retry.1s, commit past 100");
System.out.println(" offset 101: process -> OK (immediately!)");
System.out.println(" offset 102: process -> OK");
System.out.println(" Retry consumer picks up offset 100 after 1s");
}
}
Blocking retry (BAD):
offset 100: process -> FAIL
sleep 1s, retry -> FAIL
sleep 5s, retry -> FAIL
Partition STALLED for 6+ seconds
offset 101, 102, 103 waiting...
Non-blocking retry topics (GOOD):
offset 100: process -> FAIL
Publish to retry.1s, commit past 100
offset 101: process -> OK (immediately!)
offset 102: process -> OK
Retry consumer picks up offset 100 after 1s
How does the retry ladder work?
Each tier has a longer delay. A record carries a header counting attempts. Each failure increments the count and graduates the record to the next tier. When the last tier is exhausted, it goes to the DLQ.
package com.bank.retry;
public class RetryLadder {
public static void main(String[] args) {
System.out.println("Tier layout for transfers topic:");
System.out.println(" transfers (main, no delay)");
System.out.println(" transfers-retry-1s (1 second delay)");
System.out.println(" transfers-retry-5s (5 second delay)");
System.out.println(" transfers-retry-30s (30 second delay)");
System.out.println(" transfers.DLT (dead letter queue)");
System.out.println();
System.out.println("Flow for a transient failure:");
System.out.println(" main: FAIL -> publish to retry-1s, commit");
System.out.println(" retry-1s: FAIL -> publish to retry-5s");
System.out.println(" retry-5s: FAIL -> publish to retry-30s");
System.out.println(" retry-30s: FAIL -> publish to DLT");
}
}
Tier layout for transfers topic:
transfers (main, no delay)
transfers-retry-1s (1 second delay)
transfers-retry-5s (5 second delay)
transfers-retry-30s (30 second delay)
transfers.DLT (dead letter queue)
Flow for a transient failure:
main: FAIL -> publish to retry-1s, commit
retry-1s: FAIL -> publish to retry-5s
retry-5s: FAIL -> publish to retry-30s
retry-30s: FAIL -> publish to DLT
Why use exponential backoff?
Exponential backoff gives a failing dependency time to recover. Short waits first, longer waits later, so you do not hammer a struggling service.
package com.bank.retry;
public class ExponentialBackoff {
public static void main(String[] args) {
long[] delays = {1000, 5000, 30000};
long total = 0;
for (int i = 0; i < delays.length; i++) {
total += delays[i];
System.out.printf(" Attempt %d: wait %d ms (total elapsed: %d ms)%n",
i + 1, delays[i], total);
}
System.out.println(" Total retry window: " + total / 1000 + " seconds");
System.out.println(" Dependency has 36 seconds to recover");
}
}
Attempt 1: wait 1000 ms (total elapsed: 1000 ms)
Attempt 2: wait 5000 ms (total elapsed: 6000 ms)
Attempt 3: wait 30000 ms (total elapsed: 36000 ms)
Total retry window: 36 seconds
Dependency has 36 seconds to recover
Why add jitter?
Without jitter, if many records fail at the same moment because a dependency went down, they all retry at the same instant when their identical delays elapse. That thundering herd can knock the dependency over again.
package com.bank.retry;
import java.util.concurrent.ThreadLocalRandom;
public class JitterDemo {
public static long withJitter(long baseDelay) {
long jitter = (long) (baseDelay * 0.5);
return baseDelay - jitter + ThreadLocalRandom.current().nextLong(2 * jitter);
}
public static void main(String[] args) {
long baseDelay = 5000;
System.out.println("Base delay: " + baseDelay + " ms");
System.out.println("With jitter (5 samples):");
for (int i = 0; i < 5; i++) {
System.out.println(" Retry " + (i + 1) + ": " + withJitter(baseDelay) + " ms");
}
System.out.println("Retries are SPREAD -- no thundering herd");
}
}
Base delay: 5000 ms
With jitter (5 samples):
Retry 1: 3821 ms
Retry 2: 6244 ms
Retry 3: 4517 ms
Retry 4: 7012 ms
Retry 5: 2958 ms
Retries are SPREAD -- no thundering herd
How do you classify failures?
package com.bank.retry;
public class FailureClassification {
public static void main(String[] args) {
System.out.println("Transient (enter retry ladder):");
System.out.println(" - java.net.SocketTimeoutException");
System.out.println(" - org.springframework.dao.QueryTimeoutException");
System.out.println(" - HTTP 503 Service Unavailable");
System.out.println();
System.out.println("Permanent (straight to DLQ):");
System.out.println(" - IllegalArgumentException (validation)");
System.out.println(" - SerializationException (bad format)");
System.out.println(" - HTTP 400 Bad Request");
}
}
Transient (enter retry ladder):
- java.net.SocketTimeoutException
- org.springframework.dao.QueryTimeoutException
- HTTP 503 Service Unavailable
Permanent (straight to DLQ):
- IllegalArgumentException (validation)
- SerializationException (bad format)
- HTTP 400 Bad Request
How do you implement this in Spring Kafka?
package com.bank.consumer;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.TopicSuffixingStrategy;
import org.springframework.retry.annotation.Backoff;
public class RetryableTransferConsumer {
@RetryableTopic(
attempts = "4",
backoff = @Backoff(delay = 1000, multiplier = 5, maxDelay = 30000),
topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE,
exclude = {IllegalArgumentException.class}
)
@KafkaListener(topics = "transfers", groupId = "settlement-service")
public void process(String transfer) {
// Business logic -- exceptions trigger retry ladder
System.out.println("Processing: " + transfer);
}
}
What are the trade-offs?
| Aspect | Blocking retry | Retry topics |
|---|---|---|
| Main partition | Stalled during retries | Keeps flowing |
| Ordering | Preserved (same partition) | Relaxed (retried later) |
| Infrastructure | No extra topics | Extra topics per tier |
| Monitoring | Simple | Must watch each tier + DLQ |
What should a senior engineer know?
This pattern relaxes ordering because a retried record arrives later than its neighbors. Apply it to records that do not require strict ordering. Monitor every tier and the DLT. Jitter should always accompany exponential backoff in a system with correlated failures.
Visual Overview
Tiered Retry Flow
┌──────────────┐
│ transfers │ FAIL (transient)
│ (main topic) │──────────┐
└──────────────┘ │
│ commit ▼
│ ┌──────────────┐
│ │ retry-1s │ FAIL
│ │ (wait 1s) │──────────┐
│ └──────────────┘ │
│ ▼
│ ┌──────────────┐
│ │ retry-5s │ FAIL
│ │ (wait 5s) │──────┐
│ └──────────────┘ │
│ ▼
│ ┌──────────────┐
│ │ retry-30s │ FAIL
│ │ (wait 30s) │─────┐
│ └──────────────┘ │
│ ▼
│ ┌──────────────┐
│ │ transfers.DLT│
│ │ (dead letter) │
│ └──────────────┘
▼
Main partition keeps flowing!
Permanent failure: transfers ──straight──▶ transfers.DLT
Key Takeaway
Tiered retries with retry topics move the waiting off the main consumer so the partition keeps flowing. When a record fails transiently, it is published to a retry topic and the main consumer commits past it. The record graduates through increasing delays -- 1 second, 5 seconds, 30 seconds -- giving the failing dependency time to recover. If it still fails after the last tier, it lands in the dead letter queue.
Exponential backoff gives the dependency room to recover, and jitter spreads retries in time so a whole population of correlated failures does not stampede the dependency the moment it comes back. In front of the pipeline, failure classification gates which records enter the ladder: transient failures like timeouts are retried, permanent failures like validation errors skip straight to the DLQ.
At National Bank of India, the settlement consumer calls a credit bureau that sometimes times out. With tiered retry topics, the main partition never blocks during retries, failed records spread across the recovery window instead of stampeding, and permanently invalid records go straight to the DLT. Rajesh uses Spring Kafka's @RetryableTopic to auto-create the tier topics.
Remember one sentence: tiered retry topics with exponential backoff and jitter turn a flaky dependency from a partition-wide outage into a background, self-pacing recovery that protects both your throughput and the dependency itself.
---