Reprocessing dead letter queue messages safely (the redrive flow) | Kafka & Messaging

Q: What metrics should you track during redrive?

package com.bank.redrive; public class RedriveMetrics { public static void main(String[] args) { System.out.println("Redrive metrics to track:"); System.out.println(" dlt.records.total : 1500"); System.out.println(" dlt.records.replayed : 1200"); System.out.println(" dlt.records.succeeded : 1195");

Q: What should a senior engineer know?

Redrive is a deliberate, audited, ratelimited operator action, not an automatic loop. Track how many records were replayed, succeeded, and refailed. A reprocessing topic plus idempotent consumers plus a dashboard makes redrive observable and reversible. The dead letter records must carry enough meta

Definition

Redrive moves dead letter records back to the main flow under operator control, and it is safe only when consumers are idempotent and you inspect, selectively replay, or discard rather than blindly pushing everything back.

Redrive -- the controlled process of moving records from a dead letter queue back into the main topic or a reprocessing topic so that fixed consumers can handle them.
Inspection -- reading DLT records and their error metadata to classify which records should be replayed, corrected, or discarded.
Selective replay -- republishing only the records that should be reprocessed, preserving the original key, usually to a dedicated reprocessing topic.
Idempotent consumers -- consumers that produce the same result whether they see a record once or many times, keyed on a business id.
Reprocessing topic -- a dedicated topic for replayed records, isolating replay traffic from live traffic and allowing throttling.
Re-poisoning -- the anti-pattern of blindly looping DLT records back to the main topic, causing still-poisonous records to fail again.

The dead letter queue is National Bank of India's problem tray full of transfer forms that failed. Once the bug is fixed, redrive is the supervisor carefully taking forms out of the problem tray and putting them back in the normal processing line. The supervisor first reads each form to confirm it is now processable (inspect), only puts back the good ones (replay), bins the junk ones (discard), and the clerks downstream recognize I already processed this exact form so nobody is charged twice (idempotent consumers). The supervisor never dumps the whole tray back blindly.

Understanding the Concepts

What is the redrive flow?

Redrive is the controlled process of moving records from the dead letter queue back to the main processing flow so that the now-fixed consumer can handle them. It begins only after the root-cause bug is fixed and deployed.

JAVA

package com.bank.redrive;

public class RedrivePrerequisites {
    public static void main(String[] args) {
        System.out.println("Redrive prerequisites:");
        System.out.println("  1. Root cause identified");
        System.out.println("  2. Fix deployed and verified");
        System.out.println("  3. DLT records inspected and classified");
        System.out.println("  4. Replay tool configured with throttling");
        System.out.println("  5. Consumers confirmed idempotent");
    }
}

TEXT

Redrive prerequisites:
  1. Root cause identified
  2. Fix deployed and verified
  3. DLT records inspected and classified
  4. Replay tool configured with throttling
  5. Consumers confirmed idempotent

How do you inspect DLT records?

Read the DLT records and their error headers to classify each one: transient failure now resolved, permanent failure to discard, or record needing manual correction.

JAVA

package com.bank.redrive;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.header.Header;
import java.nio.charset.StandardCharsets;

public class DltInspector {
    public static void inspect(ConsumerRecord<String, String> record) {
        System.out.println("DLT Record:");
        System.out.println("  Key    : " + record.key());
        System.out.println("  Value  : " + record.value());
        for (Header header : record.headers()) {
            String value = new String(header.value(), StandardCharsets.UTF_8);
            System.out.println("  Header : " + header.key() + " = " + value);
        }
    }

    public static void main(String[] args) {
        System.out.println("Classification:");
        System.out.println("  REPLAY  : DatabaseTimeout -> fix deployed, will succeed");
        System.out.println("  DISCARD : MalformedJSON   -> garbage, cannot replay");
        System.out.println("  MANUAL  : InvalidAmount   -> needs data correction");
    }
}

TEXT

Classification:
  REPLAY  : DatabaseTimeout -> fix deployed, will succeed
  DISCARD : MalformedJSON   -> garbage, cannot replay
  MANUAL  : InvalidAmount   -> needs data correction

How do you selectively replay records?

Republish records to a dedicated reprocessing topic rather than the live main topic. Preserve the original key so partitioning and ordering semantics are respected. Throttle the replay rate.

JAVA

package com.bank.redrive;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.time.Duration;

public class RedriveReplayTool {
    public static void replay(KafkaConsumer<String, String> dltConsumer,
                               KafkaProducer<String, String> producer,
                               String reprocessTopic,
                               int throttleMs) throws InterruptedException {
        ConsumerRecords<String, String> records = dltConsumer.poll(Duration.ofSeconds(5));
        int replayed = 0;
        int discarded = 0;

        for (ConsumerRecord<String, String> record : records) {
            if (shouldReplay(record)) {
                // Preserve original key for correct partitioning
                producer.send(new ProducerRecord<>(
                    reprocessTopic, record.key(), record.value()));
                replayed++;
                Thread.sleep(throttleMs);  // Throttle
            } else {
                discarded++;
            }
        }
        dltConsumer.commitSync();
        System.out.println("Replayed: " + replayed + ", Discarded: " + discarded);
    }

    private static boolean shouldReplay(ConsumerRecord<String, String> record) {
        // Check error headers to classify
        return record.value() != null && !record.value().isEmpty();
    }

    public static void main(String[] args) {
        System.out.println("Replay to reprocessing topic (not main):");
        System.out.println("  Isolates replay traffic from live traffic");
        System.out.println("  Allows independent throttling");
        System.out.println("  Preserves original key for partition mapping");
    }
}

TEXT

Replay to reprocessing topic (not main):
  Isolates replay traffic from live traffic
  Allows independent throttling
  Preserves original key for partition mapping

Why must consumers be idempotent?

A record may have partially processed before failing. On replay, the consumer might see it again. Without idempotency, that means double debits or double fees.

JAVA

package com.bank.consumer;

import java.math.BigDecimal;
import java.util.concurrent.ConcurrentHashMap;

public class IdempotentSettlement {
    // Track processed transfer IDs
    private static final ConcurrentHashMap<String, Boolean> processedIds =
        new ConcurrentHashMap<>();

    public static void settle(String transferId, String accountId,
                               BigDecimal amount) {
        if (processedIds.putIfAbsent(transferId, true) != null) {
            System.out.println("  " + transferId + ": already settled, SKIP");
            return;
        }
        System.out.println("  " + transferId + ": settling " +
            amount + " INR for " + accountId);
    }

    public static void main(String[] args) {
        BigDecimal amount = new BigDecimal("50000.00");
        settle("TXN-001", "SAV2001", amount);  // First time
        settle("TXN-001", "SAV2001", amount);  // Replay -- skipped
        settle("TXN-002", "SAV3045", amount);  // New transfer
    }
}

TEXT

  TXN-001: settling 50000.00 INR for SAV2001
  TXN-001: already settled, SKIP
  TXN-002: settling 50000.00 INR for SAV3045

What is the re-poisoning anti-pattern?

Blindly looping the entire DLT back to the main topic causes still-poisonous records to fail again and re-fill the DLT or stall the pipeline.

JAVA

package com.bank.mistake;

public class RePoisoningAntiPattern {
    public static void main(String[] args) {
        // WRONG: blind loop
        System.out.println("WRONG: pipe ALL DLT records back to main topic");
        System.out.println("  Permanent failures fail again -> back to DLT");
        System.out.println("  Infinite loop if auto-redrive is enabled");
        System.out.println();

        // CORRECT: inspect, classify, selectively replay
        System.out.println("CORRECT: inspect, classify, selective replay");
        System.out.println("  Transient -> replay to reprocessing topic");
        System.out.println("  Permanent -> discard with audit trail");
        System.out.println("  Unknown   -> manual investigation");
    }
}

TEXT

WRONG: pipe ALL DLT records back to main topic
  Permanent failures fail again -> back to DLT
  Infinite loop if auto-redrive is enabled

CORRECT: inspect, classify, selective replay
  Transient -> replay to reprocessing topic
  Permanent -> discard with audit trail
  Unknown   -> manual investigation

What metrics should you track during redrive?

JAVA

package com.bank.redrive;

public class RedriveMetrics {
    public static void main(String[] args) {
        System.out.println("Redrive metrics to track:");
        System.out.println("  dlt.records.total       : 1500");
        System.out.println("  dlt.records.replayed    : 1200");
        System.out.println("  dlt.records.succeeded   : 1195");
        System.out.println("  dlt.records.re-failed   : 5");
        System.out.println("  dlt.records.discarded   : 300");
        System.out.println("  replay.rate.per.second  : 50");
    }
}

TEXT

Redrive metrics to track:
  dlt.records.total       : 1500
  dlt.records.replayed    : 1200
  dlt.records.succeeded   : 1195
  dlt.records.re-failed   : 5
  dlt.records.discarded   : 300
  replay.rate.per.second  : 50

What are common mistakes during redrive?

The most dangerous mistake is replaying DLT records without first confirming the bug fix is deployed, which sends them right back to the DLT. Another error is replaying the entire DLT at once without throttling, which can overwhelm the consumer group. A third mistake is not making consumers idempotent, so a record that was partially processed before failing gets duplicated on redrive.

JAVA

package com.bank.kafka;

public class RedriveMistakes {
    public static void main(String[] args) {
        System.out.println("WRONG: replay without deploying the fix");
        System.out.println("  Records go back to DLT immediately (re-poisoning)");
        System.out.println();
        System.out.println("WRONG: replay entire DLT at once");
        System.out.println("  10,000 records flood the consumer group");
        System.out.println("  Downstream services overwhelmed");
        System.out.println();
        System.out.println("WRONG: consumer not idempotent");
        System.out.println("  Record partially processed before first failure");
        System.out.println("  Redrive creates duplicate side effects");
        System.out.println();
        System.out.println("CORRECT: deploy fix, replay in small batches, idempotent consumers");
    }
}

TEXT

Expected output:
WRONG: replay without deploying the fix
  Records go back to DLT immediately (re-poisoning)

WRONG: replay entire DLT at once
  10,000 records flood the consumer group
  Downstream services overwhelmed

WRONG: consumer not idempotent
  Record partially processed before first failure
  Redrive creates duplicate side effects

CORRECT: deploy fix, replay in small batches, idempotent consumers

What should a senior engineer know?

Redrive is a deliberate, audited, rate-limited operator action, not an automatic loop. Track how many records were replayed, succeeded, and re-failed. A reprocessing topic plus idempotent consumers plus a dashboard makes redrive observable and reversible. The dead letter records must carry enough metadata to make targeted replay possible.

Visual Overview

DIAGRAM

  Redrive Flow

  ┌──────────────────┐
  │ transfers.DLT    │
  │ (dead letter)    │
  └───────┬──────────┘
          │
          ▼
  ┌──────────────────┐
  │ INSPECT          │
  │ Read records +   │
  │ error metadata   │
  └───────┬──────────┘
          │
    ┌─────┴─────┐
    │           │
    ▼           ▼
  REPLAY     DISCARD
  (fixable)  (garbage)
    │           │
    ▼           ▼
  ┌────────┐  Audit
  │ reprocess │  trail
  │ topic     │
  └────┬──────┘
       │ (throttled)
       ▼
  ┌──────────────────┐
  │ Settlement       │
  │ Consumer         │
  │ (idempotent)     │
  │ TXN-001: skip    │
  │ TXN-002: settle  │
  └──────────────────┘

Key Takeaway

Redrive is the controlled process of moving records from the dead letter queue back into the main flow after the root-cause bug is fixed. The absolute precondition is that the fix is deployed, because replaying before fixing just re-fills the DLT.

A safe redrive has three capabilities: inspection to classify each record and its error, selective replay to a dedicated reprocessing topic with the original key preserved and throttled rate, and discard with an audit trail for genuinely unprocessable records. The safety pillar is idempotent consumers, so a replayed record that was partially processed before failing is recognized and skipped rather than double-applied.

At National Bank of India, a bug once falsely rejected large transfers and sent them to transfers.DLT. After Rajesh fixed and deployed the correction, the team ran an audited, rate-limited redrive that inspected each record, replayed the processable ones to a reprocessing topic, and discarded the malformed ones. Because the settlement consumer is idempotent on the trade id, a partially-applied transfer was recognized on replay and skipped.

Remember one sentence: redrive is an explicit, observable, reversible operator action built on idempotent consumers and careful classification, not an automatic retry loop.

---