Back to Kafka Streams
Q23Question 23 of 27

How do you test Kafka Streams with TopologyTestDriver

Answer: TopologyTestDriver is a test utility that runs a Kafka Streams topology in-memory, letting you pipe input records, read output records, and inspect state stores without a running Kafka broker.

Key Concepts
  • TopologyTestDriver runs a Kafka Streams topology in-memory without a running Kafka broker.
  • TestInputTopic lets you pipe records into the topology with specific keys, values, and timestamps.
  • TestOutputTopic lets you read output records and assert on their keys, values, and order.
  • You can access and assert on state stores directly through the TopologyTestDriver.
  • Timestamp control is critical for testing windowed and join operations.
  • The kafka-streams-test-utils Maven dependency provides all test classes.
  • Separate your topology building into a static method that returns a Topology so it can be tested independently of KafkaStreams startup.

Definition

TopologyTestDriver is a test utility that runs a Kafka Streams topology in-memory, letting you pipe input records, read output records, and inspect state stores without a running Kafka broker.
  • TopologyTestDriver creates an in-memory simulation of the Kafka Streams runtime; it processes records synchronously, making tests fast and deterministic.
  • TestInputTopic is created from the driver and lets you pipe records with a specific key, value, and timestamp into a source topic of the topology.
  • TestOutputTopic is created from the driver and lets you read records from a sink topic, check their values, and verify the output queue size.
  • State store access is available through driver.getKeyValueStore(), driver.getWindowStore(), and driver.getSessionStore() for asserting on intermediate state.
  • kafka-streams-test-utils is the Maven/Gradle dependency that provides TopologyTestDriver, TestInputTopic, and TestOutputTopic.

Think of National Bank of India testing a new payment processing pipeline. Instead of sending real money through the live banking system, the test team uses a simulator that behaves exactly like the real system but runs on a single desk with paper slips. TopologyTestDriver is that simulator: you feed it paper slips (test records), it runs them through every processing station, and you check the output pile without ever touching the real bank network.

Understanding the Concepts

What is TopologyTestDriver?

TopologyTestDriver is a class from the kafka-streams-test-utils library that runs a Kafka Streams topology without a real Kafka broker. You build the topology, wrap it in a TopologyTestDriver with test properties, and then pipe records in and read records out.

The key advantage is speed and determinism. Tests run in milliseconds, not seconds. There is no network, no broker startup, no flaky timeouts. You control every record and its timestamp.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;

public class TopologyTestDriverBasic {
    public static void main(String[] args) {
        // Step 1: Build topology
        StreamsBuilder builder = new StreamsBuilder();
        builder.<String, String>stream("input").to("output");
        Topology topology = builder.build();

        // Step 2: Create test properties
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test-app");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        // Step 3: Create driver (no broker needed)
        try (TopologyTestDriver driver = new TopologyTestDriver(topology, props)) {
            System.out.println("TopologyTestDriver created successfully");
            System.out.println("  No Kafka broker running");
            System.out.println("  No network connection needed");
            System.out.println("  Tests run in-memory, fast and deterministic");
        }
    }
}
TEXT
Expected output:
TopologyTestDriver created successfully
  No Kafka broker running
  No network connection needed
  Tests run in-memory, fast and deterministic

How do you use TestInputTopic and TestOutputTopic?

TestInputTopic lets you send records into the topology. TestOutputTopic lets you read results. You create both from the TopologyTestDriver using the topic name and serializers/deserializers.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class InputOutputTopicExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> stream = builder.stream("transactions");
        stream.filter((key, value) -> value.contains("NEFT")).to("neft-transactions");

        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

        try (TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
            TestInputTopic<String, String> input = driver.createInputTopic(
                "transactions", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> output = driver.createOutputTopic(
                "neft-transactions", new StringDeserializer(), new StringDeserializer());

            input.pipeInput("SAV2001", "SAV2001|NEFT|50000|SUCCESS");
            input.pipeInput("SAV2002", "SAV2002|RTGS|75000|SUCCESS");
            input.pipeInput("CUR3001", "CUR3001|NEFT|30000|SUCCESS");

            System.out.println("Output queue size: " + output.getQueueSize());
            while (!output.isEmpty()) {
                var record = output.readKeyValue();
                System.out.println("  Key: " + record.key + ", Value: " + record.value);
            }
        }
    }
}
TEXT
Expected output:
Output queue size: 2
  Key: SAV2001, Value: SAV2001|NEFT|50000|SUCCESS
  Key: CUR3001, Value: CUR3001|NEFT|30000|SUCCESS

How do you test windowed aggregations with timestamps?

For windowed operations, you must control the record timestamps. TestInputTopic.pipeInput() accepts an Instant parameter that sets the record timestamp. This lets you place records precisely within specific windows.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.LongDeserializer;

import java.time.Duration;
import java.time.Instant;
import java.util.Properties;

public class WindowedTestExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();
        builder.<String, String>stream("atm-withdrawals")
            .groupByKey()
            .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
            .count(Materialized.as("counts"))
            .toStream()
            .selectKey((wk, v) -> wk.key())
            .mapValues((k, v) -> String.valueOf(v))
            .to("withdrawal-counts");

        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

        try (TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
            TestInputTopic<String, String> input = driver.createInputTopic(
                "atm-withdrawals", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> output = driver.createOutputTopic(
                "withdrawal-counts", new StringDeserializer(), new StringDeserializer());

            Instant base = Instant.parse("2024-01-15T10:00:00Z");

            // 3 withdrawals in window [10:00, 10:05)
            input.pipeInput("SAV2001", "w1", base);
            input.pipeInput("SAV2001", "w2", base.plusSeconds(60));
            input.pipeInput("SAV2001", "w3", base.plusSeconds(120));

            // 1 withdrawal in next window [10:05, 10:10)
            input.pipeInput("SAV2001", "w4", base.plusSeconds(360));

            System.out.println("Windowed count results:");
            while (!output.isEmpty()) {
                System.out.println("  " + output.readKeyValue().value);
            }
        }
    }
}
TEXT
Expected output:
Windowed count results:
  1
  2
  3
  1

How do you assert on state stores?

TopologyTestDriver gives direct access to state stores. After piping input records, you can read the state store to verify intermediate values. This is useful for testing aggregations where you want to check the accumulated state, not just the output.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class StateStoreAssertionExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();
        builder.<String, String>stream("transactions")
            .groupByKey()
            .count(Materialized.as("txn-count-store"));

        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

        try (TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
            TestInputTopic<String, String> input = driver.createInputTopic(
                "transactions", new StringSerializer(), new StringSerializer());

            input.pipeInput("SAV2001", "txn1");
            input.pipeInput("SAV2001", "txn2");
            input.pipeInput("CUR3001", "txn3");
            input.pipeInput("SAV2001", "txn4");

            KeyValueStore<String, Long> store =
                driver.getKeyValueStore("txn-count-store");

            System.out.println("State store assertions:");
            System.out.println("  SAV2001 count: " + store.get("SAV2001"));
            System.out.println("  CUR3001 count: " + store.get("CUR3001"));
            System.out.println("  SAV2002 count: " + store.get("SAV2002"));
        }
    }
}
TEXT
Expected output:
State store assertions:
  SAV2001 count: 3
  CUR3001 count: 1
  SAV2002 count: null

How do you structure testable topology code?

The best practice is to separate the topology definition into a static method that returns a Topology object. The main application class creates a KafkaStreams instance with this topology. Tests create a TopologyTestDriver with the same topology. This way, the topology logic is tested independently of Kafka connectivity.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class TestableTopologyDesign {

    // Topology definition: no KafkaStreams, no Properties, no side effects
    public static Topology buildTopology() {
        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> txns = builder.stream("bank-transactions");

        txns.filter((key, value) -> {
                String[] parts = value.split("\\|");
                BigDecimal amount = new BigDecimal(parts[2]);
                return amount.compareTo(new BigDecimal("100000")) > 0;
            })
            .mapValues(value -> value.toUpperCase())
            .to("high-value-alerts");

        return builder.build();
    }

    public static void main(String[] args) {
        System.out.println("Testable topology pattern:");
        System.out.println("  1. buildTopology() is a static method returning Topology");
        System.out.println("  2. Main app: new KafkaStreams(buildTopology(), props)");
        System.out.println("  3. Test: new TopologyTestDriver(buildTopology(), testProps)");
        System.out.println("  4. Same topology, different runtime");
        System.out.println();
        System.out.println("Topology:");
        System.out.println(buildTopology().describe());
    }
}
TEXT
Expected output:
Testable topology pattern:
  1. buildTopology() is a static method returning Topology
  2. Main app: new KafkaStreams(buildTopology(), props)
  3. Test: new TopologyTestDriver(buildTopology(), testProps)
  4. Same topology, different runtime

Topology:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-FILTER-0000000001
    Processor: KSTREAM-FILTER-0000000001
      --> KSTREAM-MAPVALUES-0000000002
    Processor: KSTREAM-MAPVALUES-0000000002
      --> KSTREAM-SINK-0000000003
    Sink: KSTREAM-SINK-0000000003 (topic: high-value-alerts)

What is a complete banking project with comprehensive test coverage?

This project processes loan EMI payments. The topology reads payments, validates them, calculates remaining balance, and routes results. The test class covers normal flow, edge cases, and error scenarios.

Project structure

DIAGRAM
emi-processor/
├── pom.xml
└── src/
    ├── main/
    │   └── java/
    │       └── com/
    │           └── bank/
    │               └── streams/
    │                   ├── EmiProcessorApp.java
    │                   ├── EmiProcessorTopology.java
    │                   └── EmiPayment.java
    └── test/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        └── EmiProcessorTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>emi-processorsyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>
    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

EmiPayment.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class EmiPayment {
    private String loanId;
    private String accountId;
    private BigDecimal emiAmount;
    private int installmentNumber;

    public EmiPayment() {}

    public EmiPayment(String loanId, String accountId,
                      BigDecimal emiAmount, int installmentNumber) {
        this.loanId = loanId;
        this.accountId = accountId;
        this.emiAmount = emiAmount;
        this.installmentNumber = installmentNumber;
    }

    public String getLoanId() { return loanId; }
    public String getAccountId() { return accountId; }
    public BigDecimal getEmiAmount() { return emiAmount; }
    public int getInstallmentNumber() { return installmentNumber; }

    @Override
    public String toString() {
        return loanId + "|" + accountId + "|" + emiAmount + "|" + installmentNumber;
    }

    public static EmiPayment fromString(String value) {
        String[] parts = value.split("\\|");
        return new EmiPayment(parts[0], parts[1],
            new BigDecimal(parts[2]), Integer.parseInt(parts[3]));
    }
}

EmiProcessorTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class EmiProcessorTopology {

    private static final BigDecimal MAX_EMI = new BigDecimal("500000");

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> payments = builder.stream("emi-payments");

        // Validate and route
        KStream<String, String>[] branches = payments
            .mapValues((key, value) -> {
                try {
                    EmiPayment emi = EmiPayment.fromString(value);
                    if (emi.getEmiAmount().compareTo(BigDecimal.ZERO) <= 0) {
                        return "INVALID|" + value + "|NEGATIVE_AMOUNT";
                    }
                    if (emi.getEmiAmount().compareTo(MAX_EMI) > 0) {
                        return "INVALID|" + value + "|EXCEEDS_LIMIT";
                    }
                    return "VALID|" + value;
                } catch (Exception e) {
                    return "INVALID|" + value + "|PARSE_ERROR";
                }
            })
            .branch(
                (key, value) -> value.startsWith("VALID|"),
                (key, value) -> value.startsWith("INVALID|")
            );

        branches[0].to("processed-emi");
        branches[1].to("emi-errors");

        return builder.build();
    }
}

EmiProcessorApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class EmiProcessorApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "emi-processor");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

        KafkaStreams streams = new KafkaStreams(EmiProcessorTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
        }));

        try {
            streams.start();
            System.out.println("EMI processor started");
            System.out.println("  Input:  emi-payments");
            System.out.println("  Output: processed-emi");
            System.out.println("  Errors: emi-errors");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

EmiProcessorTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class EmiProcessorTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

        try (TopologyTestDriver driver = new TopologyTestDriver(EmiProcessorTopology.build(), props)) {
            TestInputTopic<String, String> input = driver.createInputTopic(
                "emi-payments", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> validOutput = driver.createOutputTopic(
                "processed-emi", new StringDeserializer(), new StringDeserializer());
            TestOutputTopic<String, String> errorOutput = driver.createOutputTopic(
                "emi-errors", new StringDeserializer(), new StringDeserializer());

            // Test 1: Valid EMI payment
            input.pipeInput("LOAN001", "LOAN001|SAV2001|25000|6");

            // Test 2: Negative amount
            input.pipeInput("LOAN002", "LOAN002|SAV2002|-5000|3");

            // Test 3: Exceeds limit
            input.pipeInput("LOAN003", "LOAN003|CUR3001|600000|1");

            // Test 4: Parse error (bad format)
            input.pipeInput("LOAN004", "bad-data");

            // Test 5: Another valid payment
            input.pipeInput("LOAN005", "LOAN005|SAV2001|50000|12");

            System.out.println("=== Test Results ===");
            System.out.println();
            System.out.println("Valid output (" + validOutput.getQueueSize() + " records):");
            while (!validOutput.isEmpty()) {
                System.out.println("  " + validOutput.readValue());
            }
            System.out.println();
            System.out.println("Error output (" + errorOutput.getQueueSize() + " records):");
            while (!errorOutput.isEmpty()) {
                System.out.println("  " + errorOutput.readValue());
            }
            System.out.println();
            System.out.println("Test summary:");
            System.out.println("  LOAN001: valid, processed");
            System.out.println("  LOAN002: negative amount, error");
            System.out.println("  LOAN003: exceeds 500000 limit, error");
            System.out.println("  LOAN004: parse failure, error");
            System.out.println("  LOAN005: valid, processed");
        }
    }
}
TEXT
Expected output:
=== Test Results ===

Valid output (2 records):
  VALID|LOAN001|SAV2001|25000|6
  VALID|LOAN005|SAV2001|50000|12

Error output (3 records):
  INVALID|LOAN002|SAV2002|-5000|3|NEGATIVE_AMOUNT
  INVALID|LOAN003|CUR3001|600000|1|EXCEEDS_LIMIT
  INVALID|bad-data|PARSE_ERROR

Test summary:
  LOAN001: valid, processed
  LOAN002: negative amount, error
  LOAN003: exceeds 500000 limit, error
  LOAN004: parse failure, error
  LOAN005: valid, processed

How to run the project

BASH
# Step 1: Build and run tests
cd emi-processor
mvn clean package

# Step 2: Run the test class directly
java -cp target/emi-processor-1.0.0.jar:target/dependency/* com.bank.streams.EmiProcessorTopologyTest

# Step 3: For production, start Kafka and run the app
docker-compose up -d kafka zookeeper
kafka-topics.sh --create --topic emi-payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic processed-emi --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic emi-errors --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
java -cp target/emi-processor-1.0.0.jar com.bank.streams.EmiProcessorApp

What are common mistakes when testing Kafka Streams?

The most common mistake is testing with a real broker instead of using TopologyTestDriver.

Wrong: integration test with real broker for logic validation

JAVA
package com.bank.streams;

public class WrongTestApproach {
    public static void main(String[] args) {
        System.out.println("WRONG: Using real Kafka broker for unit tests");
        System.out.println();
        System.out.println("Problems:");
        System.out.println("  1. Slow: broker startup takes 5-10 seconds per test");
        System.out.println("  2. Flaky: network timeouts, port conflicts");
        System.out.println("  3. Non-deterministic: record ordering not guaranteed");
        System.out.println("  4. Heavy: requires Docker or embedded Kafka");
        System.out.println("  5. Hard to control timestamps for windowed tests");
        System.out.println();
        System.out.println("A test suite with 50 tests takes 5+ minutes.");
    }
}
TEXT
Expected output:
WRONG: Using real Kafka broker for unit tests

Problems:
  1. Slow: broker startup takes 5-10 seconds per test
  2. Flaky: network timeouts, port conflicts
  3. Non-deterministic: record ordering not guaranteed
  4. Heavy: requires Docker or embedded Kafka
  5. Hard to control timestamps for windowed tests

A test suite with 50 tests takes 5+ minutes.

Correct: TopologyTestDriver for unit tests, real broker for integration only

JAVA
package com.bank.streams;

public class CorrectTestApproach {
    public static void main(String[] args) {
        System.out.println("CORRECT: TopologyTestDriver for unit tests");
        System.out.println();
        System.out.println("Benefits:");
        System.out.println("  1. Fast: runs in milliseconds, no broker");
        System.out.println("  2. Deterministic: exact control over record order and timestamps");
        System.out.println("  3. State store access: assert on intermediate state");
        System.out.println("  4. No Docker, no network, no flakiness");
        System.out.println();
        System.out.println("Test pyramid:");
        System.out.println("  Unit tests (TopologyTestDriver):  90% of tests");
        System.out.println("  Integration tests (EmbeddedKafka): 10% of tests");
        System.out.println();
        System.out.println("A test suite with 50 unit tests takes < 5 seconds.");
    }
}
TEXT
Expected output:
CORRECT: TopologyTestDriver for unit tests

Benefits:
  1. Fast: runs in milliseconds, no broker
  2. Deterministic: exact control over record order and timestamps
  3. State store access: assert on intermediate state
  4. No Docker, no network, no flakiness

Test pyramid:
  Unit tests (TopologyTestDriver):  90% of tests
  Integration tests (EmbeddedKafka): 10% of tests

A test suite with 50 unit tests takes < 5 seconds.

What should a senior or principal engineer know?

At the senior level, testing strategy covers more than just TopologyTestDriver.

  • Topology description test: call topology.describe() and assert the topology structure. This catches accidental changes to the processing graph.
  • Serde testing: test custom serializers and deserializers independently. A bad serde is the most common source of runtime errors.
  • Multi-partition behavior: TopologyTestDriver simulates a single partition. For multi-partition behavior (repartitioning, co-partitioning), you need integration tests with EmbeddedKafkaCluster.
  • Schema evolution: test that old and new versions of records can be deserialized by the same topology.
  • Performance testing: TopologyTestDriver is not suitable for throughput testing. Use a real broker with a load generator for performance benchmarks.
  • Test coverage targets: aim for topology unit tests covering every branch path, every error case, and every window boundary.
JAVA
package com.bank.streams;

public class SeniorTestingKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal testing checklist:");
        System.out.println("  1. Test topology.describe() to catch structural changes");
        System.out.println("  2. Test custom serdes independently");
        System.out.println("  3. TopologyTestDriver = single partition only");
        System.out.println("  4. Use EmbeddedKafka for multi-partition tests");
        System.out.println("  5. Test schema evolution: old + new record versions");
        System.out.println("  6. Performance tests require real broker");
        System.out.println("  7. Cover every branch, error, and window boundary");
        System.out.println("  8. Close TopologyTestDriver in finally/try-with-resources");
    }
}
TEXT
Expected output:
Senior/Principal testing checklist:
  1. Test topology.describe() to catch structural changes
  2. Test custom serdes independently
  3. TopologyTestDriver = single partition only
  4. Use EmbeddedKafka for multi-partition tests
  5. Test schema evolution: old + new record versions
  6. Performance tests require real broker
  7. Cover every branch, error, and window boundary
  8. Close TopologyTestDriver in finally/try-with-resources

What are common interview follow-up questions?

  • Can TopologyTestDriver test joins? (Yes. Create two TestInputTopics for the left and right sides. Pipe records with specific timestamps and verify the joined output.)
  • Does TopologyTestDriver support exactly-once semantics? (No. The test driver does not simulate Kafka transactions. Exactly-once behavior requires a real broker.)
  • Can you test the Suppressed API with TopologyTestDriver? (Yes. Pipe records to fill a window, then advance stream time past the window + grace period by piping a later record. The suppressed output will appear.)
  • How do you test a custom TimestampExtractor? (Set the extractor in the test properties. Pipe records with wall-clock timestamps. The extractor will parse the event-time from the record value.)
  • What happens if you forget to close TopologyTestDriver? (The state stores remain open, which can cause resource leaks and test failures. Always use try-with-resources.)

Visual Overview

DIAGRAM
 TopologyTestDriver Workflow
 ============================

 ┌───────────────────────────────┐
 │ 1. Build Topology             │
 │    Topology t = build()       │
 └───────────────┬───────────────┘
                 │
                 ▼
 ┌───────────────────────────────┐
 │ 2. Create TopologyTestDriver  │
 │    new TopologyTestDriver(t)  │
 │    (no broker needed)         │
 └───────────────┬───────────────┘
                 │
                 ▼
 ┌───────────────────────────────┐
 │ 3. Create TestInputTopic      │
 │    driver.createInputTopic()  │
 └───────────────┬───────────────┘
                 │
                 ▼
 ┌───────────────────────────────┐
 │ 4. Pipe Input Records         │
 │    input.pipeInput(key, val)  │
 │    input.pipeInput(key, val,  │
 │                    timestamp) │
 └───────────────┬───────────────┘
                 │
                 ▼
 ┌───────────────────────────────┐
 │ 5. Read Output / Assert       │
 │    output.readValue()         │
 │    output.getQueueSize()      │
 │    driver.getKeyValueStore()  │
 └───────────────────────────────┘

Key Takeaway

TopologyTestDriver is the primary tool for testing Kafka Streams applications. It runs the entire topology in memory without a Kafka broker, making tests fast, deterministic, and reliable. You pipe records with exact timestamps, read the output, and assert on both output records and state store values.

The key to testable Kafka Streams code is separating the topology definition into a static method that returns a Topology object. The main application wraps it in KafkaStreams with production properties. Tests wrap it in TopologyTestDriver with test properties. Same topology, different runtime.

In our EMI processor project at National Bank of India, we tested five scenarios: valid payments, negative amounts, amounts exceeding the limit, parse errors, and a second valid payment. All five tests ran in a single TopologyTestDriver session, completing in milliseconds. The test verified that 2 records went to the valid output and 3 to the error output.

At the senior level, use TopologyTestDriver for 90% of tests and real brokers (EmbeddedKafka) for the remaining 10% that need multi-partition behavior. Test custom serdes independently, test topology.describe() to catch structural changes, and cover every branch path, error case, and window boundary.

Remember one sentence: TopologyTestDriver runs your topology in memory with no broker, giving you deterministic, millisecond-fast tests that control every record and timestamp.

---

Share
Use ← → arrow keys to navigate