Back to Kafka Streams
Q7Question 7 of 27

What are source, processor, and sink nodes in a Kafka Streams topology

Answer: A Kafka Streams topology is a directed graph where source nodes read from Kafka topics, processor nodes transform the data, and sink nodes write the results back to Kafka topics.

Key Concepts
  • A source node is the entry point of a topology: it reads records from one or more Kafka topics and forwards them to downstream processor nodes.
  • A processor node receives records from upstream nodes, applies transformation logic (filter, map, join, aggregate), and forwards results to downstream nodes.
  • A sink node is the exit point of a topology: it receives records from upstream processor nodes and writes them to a Kafka topic.
  • Every topology must have at least one source node and at least one sink node; processor nodes in between are optional for a pass-through pipeline.
  • The DSL (StreamsBuilder) creates these nodes automatically: stream() creates a source, filter()/map() create processors, and to() creates a sink.
  • The Processor API lets you add source, processor, and sink nodes manually with addSource(), addProcessor(), and addSink().
  • You can inspect all nodes and their connections by calling topology.describe() before starting the application.

Definition

A Kafka Streams topology is a directed graph where source nodes read from Kafka topics, processor nodes transform the data, and sink nodes write the results back to Kafka topics.
  • Source node reads records from one or more Kafka topics and pushes them into the topology; it is created by builder.stream() or builder.table() in the DSL, or topology.addSource() in the Processor API.
  • Processor node receives records from its parent node, applies logic (filter, map, join, aggregate, or custom code), and forwards results to one or more child nodes; each DSL operation like filter() or mapValues() creates a processor node.
  • Sink node receives records from its parent processor node and writes them to a Kafka output topic; it is created by .to() in the DSL or topology.addSink() in the Processor API.
  • Node graph is a directed acyclic graph (DAG) where data flows from source nodes through processor nodes to sink nodes; each node has a unique auto-generated or user-assigned name.
  • State store node is a special processor node that maintains local state (RocksDB or in-memory) for stateful operations like aggregations and joins.

Think of a cheque clearing pipeline at National Bank of India. The source node is the mailroom that receives cheques from all branches (Kafka topics). The processor nodes are the verification desk (check signature), the calculation desk (compute interest), and the fraud desk (flag suspicious amounts). The sink node is the dispatch desk that sends the processed cheques to the central ledger (output Kafka topic). Each desk receives from the previous one and passes to the next. The mailroom only receives, the dispatch desk only sends, and the desks in between do the actual work.

Understanding the Concepts

What is a source node?

A source node is the starting point of a topology. It connects to one or more Kafka topics and reads records from them. Every record that enters the topology comes through a source node.

In the DSL, builder.stream("topic") creates a source node that reads the topic as a KStream. builder.table("topic") creates a source node that reads the topic as a KTable. In the Processor API, topology.addSource("name", "topic") creates a source node explicitly.

A topology can have multiple source nodes reading from different topics.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;

public class SourceNodeDemo {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        // Source node 1: reads transactions as a stream (every event)
        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Source node 2: reads account balances as a table (latest per key)
        KTable<String, String> balances = builder.table("account-balances");

        // Source node 3: reads from multiple topics into one stream
        KStream<String, String> allAlerts = builder.stream(
            java.util.Arrays.asList("fraud-alerts", "compliance-alerts"));

        // Each creates a separate source node in the topology
        transactions.to("transactions-copy");
        allAlerts.to("all-alerts");

        Topology topology = builder.build();
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-SINK-0000000001
    Sink: KSTREAM-SINK-0000000001 (topic: transactions-copy)
      <-- KSTREAM-SOURCE-0000000000

   Sub-topology: 1
    Source: KSTREAM-SOURCE-0000000002 (topics: [fraud-alerts, compliance-alerts])
      --> KSTREAM-SINK-0000000003
    Sink: KSTREAM-SINK-0000000003 (topic: all-alerts)
      <-- KSTREAM-SOURCE-0000000002

   Sub-topology: 2
    Source: KTABLE-SOURCE-0000000004 (topics: [account-balances])
      --> KTABLE-SOURCE-0000000005
    Processor: KTABLE-SOURCE-0000000005 (stores: [account-balances-STATE-STORE-...])
      <-- KTABLE-SOURCE-0000000004

Notice that the KTable source node also creates a state store node to maintain the latest value per key.

What is a processor node?

A processor node sits between source and sink nodes. It receives records from its upstream node, applies some logic, and forwards results to its downstream node. Every transformation operation in the DSL creates a processor node.

Common processor nodes include filter (remove records), mapValues (change the value), map (change key and value), flatMap (one input produces many outputs), groupByKey (prepare for aggregation), and aggregate/reduce/count (stateful computations).

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

import java.math.BigDecimal;

public class ProcessorNodeDemo {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Processor node 1: filter (remove transactions below 10,000)
        KStream<String, String> largeOnly = transactions
            .filter((accountId, value) -> {
                BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
                return amount.compareTo(new BigDecimal("10000")) > 0;
            });

        // Processor node 2: mapValues (tag as reviewed)
        KStream<String, String> tagged = largeOnly
            .mapValues(value -> "REVIEWED|" + value.toUpperCase());

        // Processor node 3: peek (log for monitoring, does not change data)
        KStream<String, String> logged = tagged
            .peek((key, value) ->
                System.out.println("Processing: " + key + " -> " + value));

        logged.to("reviewed-transactions");

        Topology topology = builder.build();
        System.out.println("Topology with 3 processor nodes:");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Topology with 3 processor nodes:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-FILTER-0000000001
    Processor: KSTREAM-FILTER-0000000001 (stores: [])
      --> KSTREAM-MAPVALUES-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
      --> KSTREAM-PEEK-0000000003
      <-- KSTREAM-FILTER-0000000001
    Processor: KSTREAM-PEEK-0000000003 (stores: [])
      --> KSTREAM-SINK-0000000004
      <-- KSTREAM-MAPVALUES-0000000002
    Sink: KSTREAM-SINK-0000000004 (topic: reviewed-transactions)
      <-- KSTREAM-PEEK-0000000003

The topology shows three processor nodes chained together: FILTER, MAPVALUES, and PEEK. Each one feeds into the next.

What is a sink node?

A sink node is the final node in a processing path. It takes records from its upstream processor node and writes them to a Kafka output topic. In the DSL, calling .to("topic") creates a sink node.

A topology can have multiple sink nodes writing to different topics. You can also use branching to send different records to different sink nodes.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

import java.math.BigDecimal;

public class SinkNodeDemo {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Branch into two paths, each with its own sink node
        KStream<String, String> highValue = transactions
            .filter((key, value) -> {
                BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
                return amount.compareTo(new BigDecimal("100000")) > 0;
            });

        KStream<String, String> normal = transactions
            .filter((key, value) -> {
                BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
                return amount.compareTo(new BigDecimal("100000")) <= 0;
            });

        // Sink node 1: high-value transactions go to fraud review
        highValue.to("fraud-review-queue");

        // Sink node 2: normal transactions go to standard processing
        normal.to("standard-processing");

        Topology topology = builder.build();
        System.out.println("Topology with 2 sink nodes:");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Topology with 2 sink nodes:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-FILTER-0000000001, KSTREAM-FILTER-0000000002
    Processor: KSTREAM-FILTER-0000000001 (stores: [])
      --> KSTREAM-SINK-0000000003
      <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-FILTER-0000000002 (stores: [])
      --> KSTREAM-SINK-0000000004
      <-- KSTREAM-SOURCE-0000000000
    Sink: KSTREAM-SINK-0000000003 (topic: fraud-review-queue)
      <-- KSTREAM-FILTER-0000000001
    Sink: KSTREAM-SINK-0000000004 (topic: standard-processing)
      <-- KSTREAM-FILTER-0000000002

One source node fans out to two filter processor nodes, each leading to its own sink node. High-value transactions go to fraud review, normal ones go to standard processing.

How do source, processor, and sink nodes form the processing graph?

The three node types connect to form a directed acyclic graph (DAG). Data always flows in one direction: source to processor to sink. A source node can feed multiple processors, a processor can feed multiple downstream nodes, and multiple processors can feed a single sink.

Node typeCreated by (DSL)Created by (Processor API)Can receive fromCan send to
Sourcestream(), table()addSource()Kafka topic onlyProcessor or Sink
Processorfilter(), map(), join(), aggregate()addProcessor()Source or ProcessorProcessor or Sink
Sinkto()addSink()Processor onlyKafka topic only
JAVA
package com.bank.streams;

public class NodeGraphRules {
    public static void main(String[] args) {
        System.out.println("Node graph rules:");
        System.out.println();
        System.out.println("  Source node:");
        System.out.println("    - Reads from Kafka topic(s)");
        System.out.println("    - Cannot receive from other nodes");
        System.out.println("    - Sends to processor or sink nodes");
        System.out.println();
        System.out.println("  Processor node:");
        System.out.println("    - Receives from source or other processors");
        System.out.println("    - Applies transformation logic");
        System.out.println("    - Sends to other processors or sink nodes");
        System.out.println("    - May have a state store (for aggregations, joins)");
        System.out.println();
        System.out.println("  Sink node:");
        System.out.println("    - Receives from processor nodes");
        System.out.println("    - Writes to a Kafka topic");
        System.out.println("    - Cannot send to other nodes");
        System.out.println();
        System.out.println("  Data flow: Source -> Processor(s) -> Sink");
        System.out.println("  Direction: always left to right, never backwards");
    }
}
TEXT
Expected output:
Node graph rules:

  Source node:
    - Reads from Kafka topic(s)
    - Cannot receive from other nodes
    - Sends to processor or sink nodes

  Processor node:
    - Receives from source or other processors
    - Applies transformation logic
    - Sends to other processors or sink nodes
    - May have a state store (for aggregations, joins)

  Sink node:
    - Receives from processor nodes
    - Writes to a Kafka topic
    - Cannot send to other nodes

  Data flow: Source -> Processor(s) -> Sink
  Direction: always left to right, never backwards

What is a complete banking project demonstrating all three node types?

This project builds a cheque clearing service for National Bank of India. It reads cheque events, validates amounts, calculates clearing charges, flags high-value cheques for manual review, and routes results to separate output topics. The topology uses multiple source, processor, and sink nodes.

Project structure

DIAGRAM
cheque-clearing-service/
├── pom.xml
└── src/
    ├── main/
    │   └── java/
    │       └── com/
    │           └── bank/
    │               └── streams/
    │                   ├── ChequeClearingApp.java
    │                   ├── ChequeClearingTopology.java
    │                   └── ChequeEvent.java
    └── test/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        └── ChequeClearingTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>cheque-clearing-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

ChequeEvent.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class ChequeEvent {
    private String chequeId;
    private String payerAccount;
    private String payeeAccount;
    private BigDecimal amount;
    private String branch;

    public ChequeEvent() {}

    public ChequeEvent(String chequeId, String payerAccount, String payeeAccount,
                       BigDecimal amount, String branch) {
        this.chequeId = chequeId;
        this.payerAccount = payerAccount;
        this.payeeAccount = payeeAccount;
        this.amount = amount;
        this.branch = branch;
    }

    public String getChequeId() { return chequeId; }
    public String getPayerAccount() { return payerAccount; }
    public String getPayeeAccount() { return payeeAccount; }
    public BigDecimal getAmount() { return amount; }
    public String getBranch() { return branch; }

    @Override
    public String toString() {
        return chequeId + "|" + payerAccount + "|" + payeeAccount + "|" + amount + "|" + branch;
    }

    public static ChequeEvent fromString(String value) {
        String[] p = value.split("\\|");
        return new ChequeEvent(p[0], p[1], p[2], new BigDecimal(p[3]), p[4]);
    }
}

ChequeClearingTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

import java.math.BigDecimal;
import java.math.RoundingMode;

public class ChequeClearingTopology {

    private static final BigDecimal MANUAL_REVIEW_THRESHOLD = new BigDecimal("200000");
    private static final BigDecimal CLEARING_CHARGE_RATE = new BigDecimal("0.001");
    private static final BigDecimal MIN_CHARGE = new BigDecimal("25.00");

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        // SOURCE NODE: read cheque events
        KStream<String, String> cheques = builder.stream("cheque-events");

        // PROCESSOR NODE 1: validate (filter out zero or negative amounts)
        KStream<String, String> valid = cheques
            .filter((chequeId, value) -> {
                ChequeEvent evt = ChequeEvent.fromString(value);
                return evt.getAmount().compareTo(BigDecimal.ZERO) > 0;
            });

        // PROCESSOR NODE 2: calculate clearing charge
        KStream<String, String> withCharge = valid
            .mapValues(value -> {
                ChequeEvent evt = ChequeEvent.fromString(value);
                BigDecimal charge = evt.getAmount()
                    .multiply(CLEARING_CHARGE_RATE)
                    .setScale(2, RoundingMode.HALF_UP);
                if (charge.compareTo(MIN_CHARGE) < 0) {
                    charge = MIN_CHARGE;
                }
                return evt.toString() + "|CHARGE:" + charge;
            });

        // PROCESSOR NODE 3: split into high-value and normal
        KStream<String, String> highValue = withCharge
            .filter((chequeId, value) -> {
                ChequeEvent evt = ChequeEvent.fromString(value);
                return evt.getAmount().compareTo(MANUAL_REVIEW_THRESHOLD) > 0;
            });

        KStream<String, String> normal = withCharge
            .filter((chequeId, value) -> {
                ChequeEvent evt = ChequeEvent.fromString(value);
                return evt.getAmount().compareTo(MANUAL_REVIEW_THRESHOLD) <= 0;
            });

        // SINK NODE 1: high-value cheques to manual review
        highValue.to("cheque-manual-review");

        // SINK NODE 2: normal cheques to auto-clearing
        normal.to("cheque-auto-cleared");

        return builder.build();
    }
}

ChequeClearingApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class ChequeClearingApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "cheque-clearing-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        Topology topology = ChequeClearingTopology.build();
        System.out.println("Topology:");
        System.out.println(topology.describe());

        KafkaStreams streams = new KafkaStreams(topology, props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
            System.out.println("Cheque clearing service stopped");
        }));

        try {
            streams.start();
            System.out.println("Cheque clearing service started");
            System.out.println("  Input:   cheque-events");
            System.out.println("  Output:  cheque-manual-review (> 2,00,000)");
            System.out.println("  Output:  cheque-auto-cleared  (<= 2,00,000)");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

ChequeClearingTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class ChequeClearingTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(ChequeClearingTopology.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "cheque-events",
                new StringSerializer(), new StringSerializer());

            TestOutputTopic<String, String> manualReview = driver.createOutputTopic(
                "cheque-manual-review",
                new StringDeserializer(), new StringDeserializer());

            TestOutputTopic<String, String> autoCleared = driver.createOutputTopic(
                "cheque-auto-cleared",
                new StringDeserializer(), new StringDeserializer());

            // Test 1: High-value cheque (3,50,000) -> manual review
            input.pipeInput("CHQ001",
                "CHQ001|SAV2001|SAV2002|350000|Mumbai");

            // Test 2: Normal cheque (75,000) -> auto-cleared
            input.pipeInput("CHQ002",
                "CHQ002|CUR3001|SAV2003|75000|Delhi");

            // Test 3: Invalid cheque (0 amount) -> filtered out
            input.pipeInput("CHQ003",
                "CHQ003|SAV2004|SAV2005|0|Chennai");

            // Test 4: Exactly at threshold (2,00,000) -> auto-cleared
            input.pipeInput("CHQ004",
                "CHQ004|SAV2006|CUR3002|200000|Kolkata");

            System.out.println("Manual review queue: " + manualReview.getQueueSize());
            System.out.println("  " + manualReview.readValue());

            System.out.println("Auto-cleared queue: " + autoCleared.getQueueSize());
            System.out.println("  " + autoCleared.readValue());
            System.out.println("  " + autoCleared.readValue());

            System.out.println();
            System.out.println("CHQ003 (zero amount): filtered out, not in any output");
            System.out.println("Test passed: 1 manual review, 2 auto-cleared, 1 filtered");
        }
    }
}
TEXT
Expected output:
Manual review queue: 1
  CHQ001|SAV2001|SAV2002|350000|Mumbai|CHARGE:350.00
Auto-cleared queue: 2
  CHQ002|CUR3001|SAV2003|75000|Delhi|CHARGE:75.00
  CHQ004|SAV2006|CUR3002|200000|Kolkata|CHARGE:200.00

CHQ003 (zero amount): filtered out, not in any output
Test passed: 1 manual review, 2 auto-cleared, 1 filtered

How to run

BASH
# Step 1: Create topics
kafka-topics.sh --create --topic cheque-events --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic cheque-manual-review --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic cheque-auto-cleared --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092

# Step 2: Build
cd cheque-clearing-service
mvn clean package

# Step 3: Run unit test (no broker needed)
java -cp target/cheque-clearing-service-1.0.0.jar com.bank.streams.ChequeClearingTopologyTest

# Step 4: Run the application
java -cp target/cheque-clearing-service-1.0.0.jar com.bank.streams.ChequeClearingApp

# Step 5: Produce test cheques
kafka-console-producer.sh --topic cheque-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type:
# CHQ001:CHQ001|SAV2001|SAV2002|350000|Mumbai
# CHQ002:CHQ002|CUR3001|SAV2003|75000|Delhi
# CHQ003:CHQ003|SAV2004|SAV2005|0|Chennai
# CHQ004:CHQ004|SAV2006|CUR3002|200000|Kolkata

# Step 6: Consume manual review output
kafka-console-consumer.sh --topic cheque-manual-review --from-beginning --bootstrap-server localhost:9092

# Step 7: Consume auto-cleared output
kafka-console-consumer.sh --topic cheque-auto-cleared --from-beginning --bootstrap-server localhost:9092
TEXT
Expected output on cheque-manual-review:
CHQ001|SAV2001|SAV2002|350000|Mumbai|CHARGE:350.00

Expected output on cheque-auto-cleared:
CHQ002|CUR3001|SAV2003|75000|Delhi|CHARGE:75.00
CHQ004|SAV2006|CUR3002|200000|Kolkata|CHARGE:200.00

CHQ003 does not appear anywhere because the validation filter removed it.

What are common mistakes with topology nodes?

Wrong: forgetting the sink node

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;

public class WrongNoSinkNode {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        // Source and processors, but no sink
        builder.stream("bank-transactions")
            .filter((key, value) -> value.contains("SUCCESS"))
            .mapValues(String::toUpperCase);
        // Missing: .to("output-topic")

        Topology topology = builder.build();

        System.out.println("WRONG: No sink node in topology");
        System.out.println();
        System.out.println("  What happens:");
        System.out.println("    - Topology compiles without error");
        System.out.println("    - Application starts without error");
        System.out.println("    - Records are read and processed");
        System.out.println("    - But results go nowhere (silently lost)");
        System.out.println();
        System.out.println("  Fix: always end a processing path with .to() or .foreach()");
    }
}
TEXT
Expected output:
WRONG: No sink node in topology

  What happens:
    - Topology compiles without error
    - Application starts without error
    - Records are read and processed
    - But results go nowhere (silently lost)

  Fix: always end a processing path with .to() or .foreach()

Correct: every processing path ends with a sink

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;

public class CorrectWithSinkNode {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        builder.stream("bank-transactions")
            .filter((key, value) -> value.contains("SUCCESS"))
            .mapValues(String::toUpperCase)
            .to("successful-transactions");  // Sink node: results written

        Topology topology = builder.build();

        System.out.println("CORRECT: Processing path ends with .to()");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
CORRECT: Processing path ends with .to()
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-FILTER-0000000001
    Processor: KSTREAM-FILTER-0000000001 (stores: [])
      --> KSTREAM-MAPVALUES-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
      --> KSTREAM-SINK-0000000003
      <-- KSTREAM-FILTER-0000000001
    Sink: KSTREAM-SINK-0000000003 (topic: successful-transactions)
      <-- KSTREAM-MAPVALUES-0000000002

What should a senior or principal engineer know?

At the senior level, you should be able to read a topology description and trace how data flows from source through processors to sinks. You should also know when to use the Processor API for custom node behavior.

At the principal level, consider:

  • Named nodes: use Named.as("my-filter") to give nodes meaningful names instead of auto-generated ones like KSTREAM-FILTER-0000000001. This makes topology descriptions readable and ensures stability across code changes, which is critical for state store recovery.
  • Repartition nodes: when you change the key (map, selectKey, groupBy), Kafka Streams inserts an internal repartition topic and additional source/sink nodes. This is expensive. Avoid unnecessary key changes.
  • Through nodes: the deprecated through() method created an explicit intermediate topic. In modern Kafka Streams, use repartition() for explicit repartitioning control.
  • Topology testing: always call topology.describe() in tests and compare the output to a known-good baseline. If the describe() output changes unexpectedly, your state store recovery may break.
  • Fan-out cost: when a source node fans out to multiple processor branches, each branch processes every record independently. If the branches are expensive, consider filtering early to reduce work.
JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.Named;

public class NamedNodesDemo {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> txns = builder.stream("bank-transactions");

        txns
            .filter((k, v) -> v.contains("SUCCESS"),
                    Named.as("filter-successful"))
            .mapValues(String::toUpperCase,
                    Named.as("normalize-uppercase"))
            .to("processed-transactions");

        Topology topology = builder.build();
        System.out.println("Named nodes (readable topology):");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Named nodes (readable topology):
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> filter-successful
    Processor: filter-successful (stores: [])
      --> normalize-uppercase
      <-- KSTREAM-SOURCE-0000000000
    Processor: normalize-uppercase (stores: [])
      --> KSTREAM-SINK-0000000001
      <-- filter-successful
    Sink: KSTREAM-SINK-0000000001 (topic: processed-transactions)
      <-- normalize-uppercase

With Named.as(), the processor nodes show meaningful names instead of auto-generated ones. This makes debugging and monitoring much easier in production.

What are common interview follow-up questions?

  • Can a source node read from multiple topics? (Yes. Pass a list of topic names to builder.stream(Arrays.asList("topic1", "topic2")). All records from both topics flow through the same downstream processors.)
  • Can a processor node write to multiple downstream nodes? (Yes. Call multiple operations on the same stream. For example, call both .filter().to("topicA") and .mapValues().to("topicB") on the same source stream.)
  • What is a state store node? (It is a special processor node that maintains a local key-value store for stateful operations like aggregate, reduce, count, and join. It is backed by a changelog topic for fault tolerance.)
  • Can you have a topology with only a source and a sink, no processor? (Yes. builder.stream("input").to("output") creates a copy topology: source reads and sink writes, with no transformation in between.)
  • How do repartition nodes work? (When you change the key with map() or selectKey(), Kafka Streams inserts an internal sink node that writes to a repartition topic, and a new source node that reads from it. This ensures records with the same new key end up in the same partition.)
  • What happens if a processor node throws an exception? (By default, the stream thread dies. Configure streams.setUncaughtExceptionHandler() to replace the thread, shut down the client, or shut down the entire application.)

Visual Overview

DIAGRAM
 Node Types in a Topology
 =========================

 ┌──────────────────────┐
 │     SOURCE NODE       │
 │  Reads from Kafka     │
 │  topic(s)             │
 │  builder.stream()     │
 └──────────┬───────────┘
            │ records
            ▼
 ┌──────────────────────┐
 │   PROCESSOR NODE 1    │
 │  filter()             │
 │  Remove invalid       │
 └──────────┬───────────┘
            │ valid records
            ▼
 ┌──────────────────────┐
 │   PROCESSOR NODE 2    │
 │  mapValues()          │
 │  Calculate charges    │
 └──────┬──────┬────────┘
        │      │
        │      │ branch
        ▼      ▼
 ┌──────────┐ ┌──────────┐
 │ SINK 1   │ │ SINK 2   │
 │ manual   │ │ auto     │
 │ review   │ │ cleared  │
 │ .to()    │ │ .to()    │
 └──────────┘ └──────────┘


 Cheque Clearing Pipeline
 =========================

 cheque-events (topic)
       │
       ▼ Source
 ┌─────────────┐
 │ Validate    │ filter: amount > 0
 └──────┬──────┘
        ▼ Processor
 ┌─────────────┐
 │ Charge calc │ mapValues: 0.1% charge
 └──────┬──────┘
        │
   ┌────┴────┐
   ▼         ▼
 ┌─────┐  ┌─────┐
 │>200K│  │<=200K│
 │Sink │  │Sink  │
 └──┬──┘  └──┬──┘
    ▼        ▼
 manual   auto
 review   cleared

Key Takeaway

Every Kafka Streams topology is built from three types of nodes. Source nodes read records from Kafka topics and push them into the processing graph. Processor nodes receive records, apply transformation logic like filtering, mapping, joining, or aggregating, and forward the results downstream. Sink nodes receive the final results and write them to Kafka output topics.

In the DSL, these nodes are created implicitly: builder.stream() creates a source, each chained operation like filter() or mapValues() creates a processor, and .to() creates a sink. In the Processor API, you add them explicitly with addSource(), addProcessor(), and addSink(). Either way, the result is a directed graph where data flows from left to right.

In our National Bank of India cheque clearing project, one source node read cheque events, one processor validated amounts, another calculated clearing charges, two more processors branched by amount threshold, and two sink nodes routed high-value cheques to manual review and normal cheques to auto-clearing. The unit test verified all four cases: high-value, normal, zero amount (filtered out), and exactly at threshold.

At the principal level, use Named.as() to give nodes meaningful names for stable topology descriptions, watch for repartition nodes that appear when you change keys, and always end every processing path with a sink node so results are not silently lost.

Remember one sentence: a topology is a graph of source nodes that read, processor nodes that transform, and sink nodes that write, and every processing path must start at a source and end at a sink.

---

Share
Use ← → arrow keys to navigate