Definition
- DeserializationExceptionHandler is called when a record from an input topic cannot be deserialized; you choose to skip the record (CONTINUE) or shut down the thread (FAIL).
- ProductionExceptionHandler is called when Kafka Streams fails to write a record to an output or changelog topic; you choose to continue or fail.
- StreamsUncaughtExceptionHandler is called when any unhandled exception escapes user processing code; you choose to replace the thread, shut down the client, or shut down the application.
- LogAndContinueExceptionHandler is a built-in DeserializationExceptionHandler that logs the error and skips the bad record.
- Dead letter topic is a pattern where bad records are forwarded to a separate Kafka topic for later inspection and reprocessing.
Think of National Bank of India's cheque processing system. A DeserializationExceptionHandler is like the machine that scans cheques: when it encounters a torn or unreadable cheque, it can either set it aside in a reject pile (CONTINUE) or stop the entire machine (FAIL). A ProductionExceptionHandler is like the printer that stamps processed cheques: if the printer jams, the system decides whether to skip that cheque or halt. The dead letter topic is the reject pile where unreadable cheques are stored for manual review by Rajesh the developer.
Understanding the Concepts
What errors can occur in a Kafka Streams application?
Kafka Streams applications face three categories of errors:
- Deserialization errors: the input record cannot be converted from bytes to the expected type. This happens when a producer writes a String but the consumer expects JSON, or when the schema changes.
- Production errors: the application cannot write output records to Kafka. This happens during broker outages, topic authorization failures, or record-too-large errors.
- Processing errors: the user's processing logic throws an unexpected exception, such as NullPointerException, NumberFormatException, or a timeout calling an external service.
package com.bank.streams;
public class ErrorCategories {
public static void main(String[] args) {
System.out.println("Error categories in Kafka Streams:");
System.out.println();
System.out.println("1. Deserialization error:");
System.out.println(" Cause: bad data on input topic");
System.out.println(" Handler: DeserializationExceptionHandler");
System.out.println(" Config: default.deserialization.exception.handler");
System.out.println();
System.out.println("2. Production error:");
System.out.println(" Cause: cannot write to output/changelog topic");
System.out.println(" Handler: ProductionExceptionHandler");
System.out.println(" Config: default.production.exception.handler");
System.out.println();
System.out.println("3. Processing error:");
System.out.println(" Cause: unhandled exception in user code");
System.out.println(" Handler: StreamsUncaughtExceptionHandler");
System.out.println(" Config: streams.setUncaughtExceptionHandler()");
}
}
Expected output:
Error categories in Kafka Streams:
1. Deserialization error:
Cause: bad data on input topic
Handler: DeserializationExceptionHandler
Config: default.deserialization.exception.handler
2. Production error:
Cause: cannot write to output/changelog topic
Handler: ProductionExceptionHandler
Config: default.production.exception.handler
3. Processing error:
Cause: unhandled exception in user code
Handler: StreamsUncaughtExceptionHandler
Config: streams.setUncaughtExceptionHandler()
How does DeserializationExceptionHandler work?
DeserializationExceptionHandler is called when Kafka Streams cannot deserialize a record from an input topic. You implement the handler and return either DeserializationHandlerResponse.CONTINUE to skip the bad record or DeserializationHandlerResponse.FAIL to shut down the stream thread.
- Kafka Streams provides two built-in implementations:
LogAndContinueExceptionHandler: logs the error and skips the record.LogAndFailExceptionHandler: logs the error and shuts down (this is the default).
package com.bank.streams;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.streams.errors.DeserializationExceptionHandler;
import org.apache.kafka.streams.processor.ProcessorContext;
import java.util.Map;
public class BankDeserializationHandler implements DeserializationExceptionHandler {
@Override
public DeserializationHandlerResponse handle(ProcessorContext context,
ConsumerRecord<byte[], byte[]> record,
Exception exception) {
System.out.println("Deserialization error on topic: " + record.topic()
+ ", partition: " + record.partition()
+ ", offset: " + record.offset());
System.out.println(" Exception: " + exception.getMessage());
System.out.println(" Action: CONTINUE (skip bad record)");
// In production: send to dead letter topic and increment metric
return DeserializationHandlerResponse.CONTINUE;
}
@Override
public void configure(Map<String, ?> configs) {
// No configuration needed
}
public static void main(String[] args) {
System.out.println("Custom DeserializationExceptionHandler:");
System.out.println(" Logs the error details");
System.out.println(" Returns CONTINUE to skip bad records");
System.out.println(" In production: forward to dead letter topic");
System.out.println();
System.out.println("Configure in properties:");
System.out.println(" props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,");
System.out.println(" BankDeserializationHandler.class);");
}
}
Expected output:
Custom DeserializationExceptionHandler:
Logs the error details
Returns CONTINUE to skip bad records
In production: forward to dead letter topic
Configure in properties:
props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
BankDeserializationHandler.class);
How does ProductionExceptionHandler work?
ProductionExceptionHandler is called when Kafka Streams fails to write a record to an output topic or a changelog topic. You return either ProductionExceptionHandlerResponse.CONTINUE to skip the failed write or ProductionExceptionHandlerResponse.FAIL to shut down.
package com.bank.streams;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.streams.errors.ProductionExceptionHandler;
import java.util.Map;
public class BankProductionHandler implements ProductionExceptionHandler {
@Override
public ProductionExceptionHandlerResponse handle(ProducerRecord<byte[], byte[]> record,
Exception exception) {
System.out.println("Production error on topic: " + record.topic());
System.out.println(" Exception: " + exception.getMessage());
if (exception instanceof org.apache.kafka.common.errors.RecordTooLargeException) {
System.out.println(" Action: CONTINUE (skip oversized record)");
return ProductionExceptionHandlerResponse.CONTINUE;
}
System.out.println(" Action: FAIL (unknown error, shut down)");
return ProductionExceptionHandlerResponse.FAIL;
}
@Override
public void configure(Map<String, ?> configs) {
// No configuration needed
}
public static void main(String[] args) {
System.out.println("Custom ProductionExceptionHandler:");
System.out.println(" RecordTooLargeException -> CONTINUE (skip)");
System.out.println(" Other exceptions -> FAIL (shut down)");
System.out.println();
System.out.println("Configure in properties:");
System.out.println(" props.put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG,");
System.out.println(" BankProductionHandler.class);");
}
}
Expected output:
Custom ProductionExceptionHandler:
RecordTooLargeException -> CONTINUE (skip)
Other exceptions -> FAIL (shut down)
Configure in properties:
props.put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG,
BankProductionHandler.class);
How does StreamsUncaughtExceptionHandler work?
StreamsUncaughtExceptionHandler catches any unhandled exception that escapes user processing code. You set it on the KafkaStreams instance before calling start(). The handler returns one of three responses:
REPLACE_THREAD: replace the failed stream thread with a new one.SHUTDOWN_CLIENT: shut down this KafkaStreams instance but leave other instances in the group running.SHUTDOWN_APPLICATION: shut down all instances in the application (by encoding the error in the consumer group protocol).
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.errors.StreamsUncaughtExceptionHandler;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
public class UncaughtExceptionHandlerExample {
public static void main(String[] args) {
System.out.println("StreamsUncaughtExceptionHandler responses:");
System.out.println(" REPLACE_THREAD: restart the failed thread");
System.out.println(" SHUTDOWN_CLIENT: stop this instance only");
System.out.println(" SHUTDOWN_APPLICATION: stop all instances");
System.out.println();
System.out.println("Example configuration:");
System.out.println(" streams.setUncaughtExceptionHandler(exception -> {");
System.out.println(" if (exception instanceof TransientException) {");
System.out.println(" return REPLACE_THREAD;");
System.out.println(" }");
System.out.println(" return SHUTDOWN_CLIENT;");
System.out.println(" });");
System.out.println();
System.out.println("In banking: use REPLACE_THREAD for transient errors");
System.out.println("(network timeout), SHUTDOWN_CLIENT for data corruption.");
}
}
Expected output:
StreamsUncaughtExceptionHandler responses:
REPLACE_THREAD: restart the failed thread
SHUTDOWN_CLIENT: stop this instance only
SHUTDOWN_APPLICATION: stop all instances
Example configuration:
streams.setUncaughtExceptionHandler(exception -> {
if (exception instanceof TransientException) {
return REPLACE_THREAD;
}
return SHUTDOWN_CLIENT;
});
In banking: use REPLACE_THREAD for transient errors
(network timeout), SHUTDOWN_CLIENT for data corruption.
How do the three error handlers compare?
| Feature | DeserializationExceptionHandler | ProductionExceptionHandler | StreamsUncaughtExceptionHandler |
|---|---|---|---|
| When called | Cannot deserialize input record | Cannot write to output/changelog | Unhandled exception in user code |
| Configuration | StreamsConfig property | StreamsConfig property | streams.setUncaughtExceptionHandler() |
| Options | CONTINUE, FAIL | CONTINUE, FAIL | REPLACE_THREAD, SHUTDOWN_CLIENT, SHUTDOWN_APPLICATION |
| Default behavior | LogAndFail (shuts down) | LogAndFail (shuts down) | Shuts down client |
| Access to record | Yes (ConsumerRecord) | Yes (ProducerRecord) | No (only exception) |
| Can send to DLT | Yes (in handler) | Yes (in handler) | Difficult (no record context) |
package com.bank.streams;
public class ErrorHandlerComparison {
public static void main(String[] args) {
System.out.println("Error handler comparison:");
System.out.println();
System.out.println("DeserializationExceptionHandler:");
System.out.println(" Trigger: bad input data");
System.out.println(" Example: producer sends XML, deserializer expects JSON");
System.out.println(" Best action: CONTINUE + send to dead letter topic");
System.out.println();
System.out.println("ProductionExceptionHandler:");
System.out.println(" Trigger: cannot write output");
System.out.println(" Example: record exceeds max.message.bytes");
System.out.println(" Best action: CONTINUE for size errors, FAIL for auth errors");
System.out.println();
System.out.println("StreamsUncaughtExceptionHandler:");
System.out.println(" Trigger: bug in processing code");
System.out.println(" Example: NullPointerException in map function");
System.out.println(" Best action: REPLACE_THREAD for transient, SHUTDOWN_CLIENT for permanent");
}
}
Expected output:
Error handler comparison:
DeserializationExceptionHandler:
Trigger: bad input data
Example: producer sends XML, deserializer expects JSON
Best action: CONTINUE + send to dead letter topic
ProductionExceptionHandler:
Trigger: cannot write output
Example: record exceeds max.message.bytes
Best action: CONTINUE for size errors, FAIL for auth errors
StreamsUncaughtExceptionHandler:
Trigger: bug in processing code
Example: NullPointerException in map function
Best action: REPLACE_THREAD for transient, SHUTDOWN_CLIENT for permanent
What is the dead letter topic pattern?
A dead letter topic (DLT) is a separate Kafka topic where you forward records that cannot be processed. Instead of dropping bad records, you write them to the DLT with metadata about the error. A separate consumer can read the DLT for manual review or reprocessing.
This is critical in banking: you cannot silently lose transactions. Every failed record must be traceable.
package com.bank.streams;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class DeadLetterTopicPattern {
static KafkaProducer<String, String> createDltProducer() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
return new KafkaProducer<>(props);
}
static void sendToDeadLetter(KafkaProducer<String, String> producer,
String originalTopic, String key,
String value, String error) {
String dltValue = "ERROR|topic=" + originalTopic
+ "|key=" + key
+ "|value=" + value
+ "|reason=" + error;
producer.send(new ProducerRecord<>(originalTopic + "-dlt", key, dltValue));
}
public static void main(String[] args) {
System.out.println("Dead letter topic pattern:");
System.out.println(" Input topic: bank-transactions");
System.out.println(" DLT topic: bank-transactions-dlt");
System.out.println();
System.out.println(" Record: SAV2001|INVALID_AMOUNT|abc");
System.out.println(" Error: NumberFormatException on amount field");
System.out.println();
System.out.println(" DLT record:");
System.out.println(" ERROR|topic=bank-transactions|key=SAV2001");
System.out.println(" |value=SAV2001|INVALID_AMOUNT|abc");
System.out.println(" |reason=NumberFormatException: For input string abc");
}
}
Expected output:
Dead letter topic pattern:
Input topic: bank-transactions
DLT topic: bank-transactions-dlt
Record: SAV2001|INVALID_AMOUNT|abc
Error: NumberFormatException on amount field
DLT record:
ERROR|topic=bank-transactions|key=SAV2001
|value=SAV2001|INVALID_AMOUNT|abc
|reason=NumberFormatException: For input string abc
What is a complete banking project for resilient transaction processing?
This project reads bank transactions, parses them, and routes valid transactions to an output topic. Invalid records that fail parsing are sent to a dead letter topic. The application uses all three error handlers.
Project structure
resilient-processor/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── ResilientProcessorApp.java
│ ├── ResilientProcessorTopology.java
│ ├── TransactionParser.java
│ └── BankDeserializationHandler.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── ResilientProcessorTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>resilient-processorsyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
TransactionParser.java
package com.bank.streams;
import java.math.BigDecimal;
public class TransactionParser {
public static String parseAndValidate(String key, String value) {
try {
String[] parts = value.split("\\|");
if (parts.length < 4) {
return "DLT|" + key + "|" + value + "|INVALID_FORMAT";
}
String accountId = parts[0];
String type = parts[1];
BigDecimal amount = new BigDecimal(parts[2]);
String status = parts[3];
if (amount.compareTo(BigDecimal.ZERO) <= 0) {
return "DLT|" + key + "|" + value + "|NEGATIVE_AMOUNT";
}
return "VALID|" + accountId + "|" + type + "|" + amount + "|" + status;
} catch (NumberFormatException e) {
return "DLT|" + key + "|" + value + "|" + e.getMessage();
}
}
public static void main(String[] args) {
System.out.println("Valid: " + parseAndValidate("SAV2001", "SAV2001|NEFT|50000|SUCCESS"));
System.out.println("Invalid: " + parseAndValidate("SAV2002", "SAV2002|NEFT|abc|SUCCESS"));
System.out.println("Short: " + parseAndValidate("SAV2003", "SAV2003|NEFT"));
System.out.println("Neg: " + parseAndValidate("SAV2004", "SAV2004|NEFT|-500|SUCCESS"));
}
}
Expected output:
Valid: VALID|SAV2001|NEFT|50000|SUCCESS
Invalid: DLT|SAV2002|SAV2002|NEFT|abc|SUCCESS|For input string: "abc"
Short: DLT|SAV2003|SAV2003|NEFT|INVALID_FORMAT
Neg: DLT|SAV2004|SAV2004|NEFT|-500|SUCCESS|NEGATIVE_AMOUNT
ResilientProcessorTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class ResilientProcessorTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Parse and validate each record
KStream<String, String> parsed = transactions.mapValues(
(key, value) -> TransactionParser.parseAndValidate(key, value));
// Branch: valid records go to output, invalid to DLT
KStream<String, String>[] branches = parsed.branch(
(key, value) -> value.startsWith("VALID|"),
(key, value) -> value.startsWith("DLT|")
);
branches[0].to("processed-transactions");
branches[1].to("bank-transactions-dlt");
return builder.build();
}
}
ResilientProcessorApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.errors.StreamsUncaughtExceptionHandler;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class ResilientProcessorApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "resilient-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
"org.apache.kafka.streams.errors.LogAndContinueExceptionHandler");
KafkaStreams streams = new KafkaStreams(
ResilientProcessorTopology.build(), props);
// Handle uncaught exceptions: replace thread for transient errors
streams.setUncaughtExceptionHandler(exception -> {
System.out.println("Uncaught exception: " + exception.getMessage());
return StreamsUncaughtExceptionHandler
.StreamThreadExceptionResponse.REPLACE_THREAD;
});
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
}));
try {
streams.start();
System.out.println("Resilient processor started");
System.out.println(" Input: bank-transactions");
System.out.println(" Output: processed-transactions");
System.out.println(" DLT: bank-transactions-dlt");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
ResilientProcessorTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class ResilientProcessorTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(ResilientProcessorTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"bank-transactions", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> validOutput = driver.createOutputTopic(
"processed-transactions",
new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> dltOutput = driver.createOutputTopic(
"bank-transactions-dlt",
new StringDeserializer(), new StringDeserializer());
// Valid record
input.pipeInput("SAV2001", "SAV2001|NEFT|50000|SUCCESS");
// Invalid: bad amount
input.pipeInput("SAV2002", "SAV2002|NEFT|abc|SUCCESS");
// Invalid: too few fields
input.pipeInput("SAV2003", "SAV2003|NEFT");
// Valid record
input.pipeInput("CUR3001", "CUR3001|RTGS|75000|SUCCESS");
System.out.println("Valid output:");
while (!validOutput.isEmpty()) {
System.out.println(" " + validOutput.readValue());
}
System.out.println();
System.out.println("Dead letter topic:");
while (!dltOutput.isEmpty()) {
System.out.println(" " + dltOutput.readValue());
}
}
}
}
Expected output:
Valid output:
VALID|SAV2001|NEFT|50000|SUCCESS
VALID|CUR3001|RTGS|75000|SUCCESS
Dead letter topic:
DLT|SAV2002|SAV2002|NEFT|abc|SUCCESS|For input string: "abc"
DLT|SAV2003|SAV2003|NEFT|INVALID_FORMAT
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics
kafka-topics.sh --create --topic bank-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic processed-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic bank-transactions-dlt --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build
cd resilient-processor
mvn clean package
# Step 4: Run
java -cp target/resilient-processor-1.0.0.jar com.bank.streams.ResilientProcessorApp
# Step 5: Produce test messages
kafka-console-producer.sh --topic bank-transactions --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|NEFT|50000|SUCCESS
# SAV2002:SAV2002|NEFT|abc|SUCCESS
# Step 6: Check valid output
kafka-console-consumer.sh --topic processed-transactions --from-beginning --bootstrap-server localhost:9092
# Step 7: Check dead letter topic
kafka-console-consumer.sh --topic bank-transactions-dlt --from-beginning --bootstrap-server localhost:9092
What are common mistakes with error handling in Kafka Streams?
The most common mistake is using the default handler (LogAndFail) in production, which shuts down the entire application on a single bad record.
Wrong: default handler shuts down on one bad record
package com.bank.streams;
public class WrongDefaultHandler {
public static void main(String[] args) {
System.out.println("WRONG: Using default LogAndFailExceptionHandler in production");
System.out.println();
System.out.println("Scenario:");
System.out.println(" 1. Application processes 10,000 transactions per second");
System.out.println(" 2. One producer sends a malformed record");
System.out.println(" 3. Default handler: LogAndFailExceptionHandler");
System.out.println(" 4. Result: ENTIRE APPLICATION SHUTS DOWN");
System.out.println();
System.out.println("Impact: All 10,000 transactions/second stop processing.");
System.out.println("One bad record from one producer halts the entire bank.");
}
}
Expected output:
WRONG: Using default LogAndFailExceptionHandler in production
Scenario:
1. Application processes 10,000 transactions per second
2. One producer sends a malformed record
3. Default handler: LogAndFailExceptionHandler
4. Result: ENTIRE APPLICATION SHUTS DOWN
Impact: All 10,000 transactions/second stop processing.
One bad record from one producer halts the entire bank.
Correct: custom handler with dead letter topic and alerting
package com.bank.streams;
public class CorrectErrorHandling {
public static void main(String[] args) {
System.out.println("CORRECT: Custom handler with DLT and alerting");
System.out.println();
System.out.println("Scenario:");
System.out.println(" 1. Application processes 10,000 transactions per second");
System.out.println(" 2. One producer sends a malformed record");
System.out.println(" 3. Custom handler: log, send to DLT, increment metric, CONTINUE");
System.out.println(" 4. Result: bad record in DLT, processing continues");
System.out.println();
System.out.println("Monitoring:");
System.out.println(" Alert on DLT consumer lag > 0 (new bad records)");
System.out.println(" Dashboard shows bad record count by producer");
System.out.println(" Manual review team processes DLT records");
}
}
Expected output:
CORRECT: Custom handler with DLT and alerting
Scenario:
1. Application processes 10,000 transactions per second
2. One producer sends a malformed record
3. Custom handler: log, send to DLT, increment metric, CONTINUE
4. Result: bad record in DLT, processing continues
Monitoring:
Alert on DLT consumer lag > 0 (new bad records)
Dashboard shows bad record count by producer
Manual review team processes DLT records
What should a senior or principal engineer know?
At the senior level, error handling strategy is a production readiness requirement.
- Handler combination: use all three handlers together. DeserializationExceptionHandler for bad input, ProductionExceptionHandler for write failures, StreamsUncaughtExceptionHandler for bugs in processing code.
- DLT schema: include the original topic, partition, offset, timestamp, key, value, and exception message in the DLT record. This makes reprocessing possible.
- Retry vs skip: for transient errors (network timeout), REPLACE_THREAD retries from the last committed offset. For permanent errors (bad data), CONTINUE skips the record.
- Exactly-once and errors: when using exactly_once_v2, a failed production handler that returns CONTINUE can leave the transaction in an inconsistent state. Test this scenario carefully.
- Monitoring: export custom metrics for deserialization errors, production errors, and DLT record counts. Alert on thresholds (e.g., more than 10 DLT records per minute).
- Circuit breaker: if the DLT rate exceeds a threshold, it may indicate a systematic problem (schema change, broken producer). Consider shutting down to prevent cascading data loss.
package com.bank.streams;
public class SeniorErrorHandlingKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal error handling checklist:");
System.out.println(" 1. Use all three handlers: deser, production, uncaught");
System.out.println(" 2. DLT records must include: topic, partition, offset, error");
System.out.println(" 3. REPLACE_THREAD for transient, CONTINUE for permanent");
System.out.println(" 4. Test error handlers with exactly_once_v2");
System.out.println(" 5. Export custom metrics for each error type");
System.out.println(" 6. Alert on DLT rate > threshold (schema change?)");
System.out.println(" 7. Circuit breaker: shut down if DLT rate too high");
System.out.println(" 8. Reprocessing pipeline: DLT -> fix -> replay");
}
}
Expected output:
Senior/Principal error handling checklist:
1. Use all three handlers: deser, production, uncaught
2. DLT records must include: topic, partition, offset, error
3. REPLACE_THREAD for transient, CONTINUE for permanent
4. Test error handlers with exactly_once_v2
5. Export custom metrics for each error type
6. Alert on DLT rate > threshold (schema change?)
7. Circuit breaker: shut down if DLT rate too high
8. Reprocessing pipeline: DLT -> fix -> replay
What are common interview follow-up questions?
- What is the default DeserializationExceptionHandler? (LogAndFailExceptionHandler. It logs the error and shuts down the stream thread. This is safe for development but dangerous in production.)
- Can you use different handlers for different topics? (No. The handler is set globally via StreamsConfig. You can check the topic name inside the handler to apply different logic per topic.)
- What happens to uncommitted records when a thread is replaced? (Processing restarts from the last committed offset. Records processed but not yet committed are reprocessed.)
- How do you reprocess DLT records? (Read from the DLT topic, fix the records, and write them back to the original input topic. Use a separate consumer group for the DLT reader.)
- Does the DeserializationExceptionHandler catch schema registry errors? (Yes. If the Avro deserializer fails because the schema is not in the registry, the handler catches it.)
Visual Overview
Error Handling Flow in Kafka Streams
=====================================
┌────────────────┐
│ Input Topic │
│ (raw records) │
└───────┬────────┘
│
▼
┌───────────────────────────────┐
│ Deserialize │
│ Success? ──────────────────────> Process
│ Fail? ──> DeserExHandler │
│ ├─ CONTINUE (skip)│
│ └─ FAIL (shutdown)│
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ Process (user code) │
│ Success? ──────────────────────> Produce
│ Exception? ──> Uncaught │
│ Handler │
│ ├─ REPLACE_THREAD │
│ ├─ SHUTDOWN_CLIENT │
│ └─ SHUTDOWN_APPLICATION │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ Produce to output topic │
│ Success? ──────────────────────> Done
│ Fail? ──> ProdExHandler │
│ ├─ CONTINUE (skip)│
│ └─ FAIL (shutdown)│
└───────────────────────────────┘
Dead Letter Topic Pattern
==========================
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Input │───>│ Process │───>│ Output │
│ Topic │ │ │ │ Topic │
└──────────┘ └────┬─────┘ └──────────┘
│ error
▼
┌──────────┐
│ Dead │
│ Letter │
│ Topic │
└──────────┘
Key Takeaway
Kafka Streams provides three layers of error handling. DeserializationExceptionHandler catches bad input records before processing starts. ProductionExceptionHandler catches failures when writing to output topics. StreamsUncaughtExceptionHandler catches any unhandled exception from your processing code. Together, they form a complete error handling strategy.
The default behavior for all handlers is to shut down on error. This is safe for development but dangerous in production. A single malformed record from one producer should not halt the entire transaction processing system at National Bank of India.
In our resilient processor project, we used the branch pattern to route valid records to the output topic and invalid records to a dead letter topic. The DLT record includes the original data and the error reason, making it possible for the operations team to review, fix, and reprocess failed records.
At the senior level, combine all three handlers, include full metadata in DLT records (topic, partition, offset, error), export custom metrics for each error type, and set up alerting when the DLT rate exceeds a threshold. A spike in DLT records often indicates a schema change or a broken producer, not just bad data.
Remember one sentence: use LogAndContinue for deserialization errors, a dead letter topic for traceability, and REPLACE_THREAD for transient processing failures to keep the banking pipeline running while capturing every failed record.
---