Definition
- Source node reads records from one or more Kafka topics and pushes them into the topology; it is created by
builder.stream()orbuilder.table()in the DSL, ortopology.addSource()in the Processor API. - Processor node receives records from its parent node, applies logic (filter, map, join, aggregate, or custom code), and forwards results to one or more child nodes; each DSL operation like
filter()ormapValues()creates a processor node. - Sink node receives records from its parent processor node and writes them to a Kafka output topic; it is created by
.to()in the DSL ortopology.addSink()in the Processor API. - Node graph is a directed acyclic graph (DAG) where data flows from source nodes through processor nodes to sink nodes; each node has a unique auto-generated or user-assigned name.
- State store node is a special processor node that maintains local state (RocksDB or in-memory) for stateful operations like aggregations and joins.
Think of a cheque clearing pipeline at National Bank of India. The source node is the mailroom that receives cheques from all branches (Kafka topics). The processor nodes are the verification desk (check signature), the calculation desk (compute interest), and the fraud desk (flag suspicious amounts). The sink node is the dispatch desk that sends the processed cheques to the central ledger (output Kafka topic). Each desk receives from the previous one and passes to the next. The mailroom only receives, the dispatch desk only sends, and the desks in between do the actual work.
Understanding the Concepts
What is a source node?
A source node is the starting point of a topology. It connects to one or more Kafka topics and reads records from them. Every record that enters the topology comes through a source node.
In the DSL, builder.stream("topic") creates a source node that reads the topic as a KStream. builder.table("topic") creates a source node that reads the topic as a KTable. In the Processor API, topology.addSource("name", "topic") creates a source node explicitly.
A topology can have multiple source nodes reading from different topics.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
public class SourceNodeDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Source node 1: reads transactions as a stream (every event)
KStream<String, String> transactions = builder.stream("bank-transactions");
// Source node 2: reads account balances as a table (latest per key)
KTable<String, String> balances = builder.table("account-balances");
// Source node 3: reads from multiple topics into one stream
KStream<String, String> allAlerts = builder.stream(
java.util.Arrays.asList("fraud-alerts", "compliance-alerts"));
// Each creates a separate source node in the topology
transactions.to("transactions-copy");
allAlerts.to("all-alerts");
Topology topology = builder.build();
System.out.println(topology.describe());
}
}
Expected output:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> KSTREAM-SINK-0000000001
Sink: KSTREAM-SINK-0000000001 (topic: transactions-copy)
<-- KSTREAM-SOURCE-0000000000
Sub-topology: 1
Source: KSTREAM-SOURCE-0000000002 (topics: [fraud-alerts, compliance-alerts])
--> KSTREAM-SINK-0000000003
Sink: KSTREAM-SINK-0000000003 (topic: all-alerts)
<-- KSTREAM-SOURCE-0000000002
Sub-topology: 2
Source: KTABLE-SOURCE-0000000004 (topics: [account-balances])
--> KTABLE-SOURCE-0000000005
Processor: KTABLE-SOURCE-0000000005 (stores: [account-balances-STATE-STORE-...])
<-- KTABLE-SOURCE-0000000004
Notice that the KTable source node also creates a state store node to maintain the latest value per key.
What is a processor node?
A processor node sits between source and sink nodes. It receives records from its upstream node, applies some logic, and forwards results to its downstream node. Every transformation operation in the DSL creates a processor node.
Common processor nodes include filter (remove records), mapValues (change the value), map (change key and value), flatMap (one input produces many outputs), groupByKey (prepare for aggregation), and aggregate/reduce/count (stateful computations).
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
public class ProcessorNodeDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Processor node 1: filter (remove transactions below 10,000)
KStream<String, String> largeOnly = transactions
.filter((accountId, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
return amount.compareTo(new BigDecimal("10000")) > 0;
});
// Processor node 2: mapValues (tag as reviewed)
KStream<String, String> tagged = largeOnly
.mapValues(value -> "REVIEWED|" + value.toUpperCase());
// Processor node 3: peek (log for monitoring, does not change data)
KStream<String, String> logged = tagged
.peek((key, value) ->
System.out.println("Processing: " + key + " -> " + value));
logged.to("reviewed-transactions");
Topology topology = builder.build();
System.out.println("Topology with 3 processor nodes:");
System.out.println(topology.describe());
}
}
Expected output:
Topology with 3 processor nodes:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-MAPVALUES-0000000002
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
--> KSTREAM-PEEK-0000000003
<-- KSTREAM-FILTER-0000000001
Processor: KSTREAM-PEEK-0000000003 (stores: [])
--> KSTREAM-SINK-0000000004
<-- KSTREAM-MAPVALUES-0000000002
Sink: KSTREAM-SINK-0000000004 (topic: reviewed-transactions)
<-- KSTREAM-PEEK-0000000003
The topology shows three processor nodes chained together: FILTER, MAPVALUES, and PEEK. Each one feeds into the next.
What is a sink node?
A sink node is the final node in a processing path. It takes records from its upstream processor node and writes them to a Kafka output topic. In the DSL, calling .to("topic") creates a sink node.
A topology can have multiple sink nodes writing to different topics. You can also use branching to send different records to different sink nodes.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
public class SinkNodeDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Branch into two paths, each with its own sink node
KStream<String, String> highValue = transactions
.filter((key, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
return amount.compareTo(new BigDecimal("100000")) > 0;
});
KStream<String, String> normal = transactions
.filter((key, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
return amount.compareTo(new BigDecimal("100000")) <= 0;
});
// Sink node 1: high-value transactions go to fraud review
highValue.to("fraud-review-queue");
// Sink node 2: normal transactions go to standard processing
normal.to("standard-processing");
Topology topology = builder.build();
System.out.println("Topology with 2 sink nodes:");
System.out.println(topology.describe());
}
}
Expected output:
Topology with 2 sink nodes:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> KSTREAM-FILTER-0000000001, KSTREAM-FILTER-0000000002
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-SINK-0000000003
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-FILTER-0000000002 (stores: [])
--> KSTREAM-SINK-0000000004
<-- KSTREAM-SOURCE-0000000000
Sink: KSTREAM-SINK-0000000003 (topic: fraud-review-queue)
<-- KSTREAM-FILTER-0000000001
Sink: KSTREAM-SINK-0000000004 (topic: standard-processing)
<-- KSTREAM-FILTER-0000000002
One source node fans out to two filter processor nodes, each leading to its own sink node. High-value transactions go to fraud review, normal ones go to standard processing.
How do source, processor, and sink nodes form the processing graph?
The three node types connect to form a directed acyclic graph (DAG). Data always flows in one direction: source to processor to sink. A source node can feed multiple processors, a processor can feed multiple downstream nodes, and multiple processors can feed a single sink.
| Node type | Created by (DSL) | Created by (Processor API) | Can receive from | Can send to |
|---|---|---|---|---|
| Source | stream(), table() | addSource() | Kafka topic only | Processor or Sink |
| Processor | filter(), map(), join(), aggregate() | addProcessor() | Source or Processor | Processor or Sink |
| Sink | to() | addSink() | Processor only | Kafka topic only |
package com.bank.streams;
public class NodeGraphRules {
public static void main(String[] args) {
System.out.println("Node graph rules:");
System.out.println();
System.out.println(" Source node:");
System.out.println(" - Reads from Kafka topic(s)");
System.out.println(" - Cannot receive from other nodes");
System.out.println(" - Sends to processor or sink nodes");
System.out.println();
System.out.println(" Processor node:");
System.out.println(" - Receives from source or other processors");
System.out.println(" - Applies transformation logic");
System.out.println(" - Sends to other processors or sink nodes");
System.out.println(" - May have a state store (for aggregations, joins)");
System.out.println();
System.out.println(" Sink node:");
System.out.println(" - Receives from processor nodes");
System.out.println(" - Writes to a Kafka topic");
System.out.println(" - Cannot send to other nodes");
System.out.println();
System.out.println(" Data flow: Source -> Processor(s) -> Sink");
System.out.println(" Direction: always left to right, never backwards");
}
}
Expected output:
Node graph rules:
Source node:
- Reads from Kafka topic(s)
- Cannot receive from other nodes
- Sends to processor or sink nodes
Processor node:
- Receives from source or other processors
- Applies transformation logic
- Sends to other processors or sink nodes
- May have a state store (for aggregations, joins)
Sink node:
- Receives from processor nodes
- Writes to a Kafka topic
- Cannot send to other nodes
Data flow: Source -> Processor(s) -> Sink
Direction: always left to right, never backwards
What is a complete banking project demonstrating all three node types?
This project builds a cheque clearing service for National Bank of India. It reads cheque events, validates amounts, calculates clearing charges, flags high-value cheques for manual review, and routes results to separate output topics. The topology uses multiple source, processor, and sink nodes.
Project structure
cheque-clearing-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── ChequeClearingApp.java
│ ├── ChequeClearingTopology.java
│ └── ChequeEvent.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── ChequeClearingTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>cheque-clearing-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
ChequeEvent.java
package com.bank.streams;
import java.math.BigDecimal;
public class ChequeEvent {
private String chequeId;
private String payerAccount;
private String payeeAccount;
private BigDecimal amount;
private String branch;
public ChequeEvent() {}
public ChequeEvent(String chequeId, String payerAccount, String payeeAccount,
BigDecimal amount, String branch) {
this.chequeId = chequeId;
this.payerAccount = payerAccount;
this.payeeAccount = payeeAccount;
this.amount = amount;
this.branch = branch;
}
public String getChequeId() { return chequeId; }
public String getPayerAccount() { return payerAccount; }
public String getPayeeAccount() { return payeeAccount; }
public BigDecimal getAmount() { return amount; }
public String getBranch() { return branch; }
@Override
public String toString() {
return chequeId + "|" + payerAccount + "|" + payeeAccount + "|" + amount + "|" + branch;
}
public static ChequeEvent fromString(String value) {
String[] p = value.split("\\|");
return new ChequeEvent(p[0], p[1], p[2], new BigDecimal(p[3]), p[4]);
}
}
ChequeClearingTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
import java.math.RoundingMode;
public class ChequeClearingTopology {
private static final BigDecimal MANUAL_REVIEW_THRESHOLD = new BigDecimal("200000");
private static final BigDecimal CLEARING_CHARGE_RATE = new BigDecimal("0.001");
private static final BigDecimal MIN_CHARGE = new BigDecimal("25.00");
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// SOURCE NODE: read cheque events
KStream<String, String> cheques = builder.stream("cheque-events");
// PROCESSOR NODE 1: validate (filter out zero or negative amounts)
KStream<String, String> valid = cheques
.filter((chequeId, value) -> {
ChequeEvent evt = ChequeEvent.fromString(value);
return evt.getAmount().compareTo(BigDecimal.ZERO) > 0;
});
// PROCESSOR NODE 2: calculate clearing charge
KStream<String, String> withCharge = valid
.mapValues(value -> {
ChequeEvent evt = ChequeEvent.fromString(value);
BigDecimal charge = evt.getAmount()
.multiply(CLEARING_CHARGE_RATE)
.setScale(2, RoundingMode.HALF_UP);
if (charge.compareTo(MIN_CHARGE) < 0) {
charge = MIN_CHARGE;
}
return evt.toString() + "|CHARGE:" + charge;
});
// PROCESSOR NODE 3: split into high-value and normal
KStream<String, String> highValue = withCharge
.filter((chequeId, value) -> {
ChequeEvent evt = ChequeEvent.fromString(value);
return evt.getAmount().compareTo(MANUAL_REVIEW_THRESHOLD) > 0;
});
KStream<String, String> normal = withCharge
.filter((chequeId, value) -> {
ChequeEvent evt = ChequeEvent.fromString(value);
return evt.getAmount().compareTo(MANUAL_REVIEW_THRESHOLD) <= 0;
});
// SINK NODE 1: high-value cheques to manual review
highValue.to("cheque-manual-review");
// SINK NODE 2: normal cheques to auto-clearing
normal.to("cheque-auto-cleared");
return builder.build();
}
}
ChequeClearingApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class ChequeClearingApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "cheque-clearing-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
Topology topology = ChequeClearingTopology.build();
System.out.println("Topology:");
System.out.println(topology.describe());
KafkaStreams streams = new KafkaStreams(topology, props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Cheque clearing service stopped");
}));
try {
streams.start();
System.out.println("Cheque clearing service started");
System.out.println(" Input: cheque-events");
System.out.println(" Output: cheque-manual-review (> 2,00,000)");
System.out.println(" Output: cheque-auto-cleared (<= 2,00,000)");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
ChequeClearingTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class ChequeClearingTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(ChequeClearingTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"cheque-events",
new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> manualReview = driver.createOutputTopic(
"cheque-manual-review",
new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> autoCleared = driver.createOutputTopic(
"cheque-auto-cleared",
new StringDeserializer(), new StringDeserializer());
// Test 1: High-value cheque (3,50,000) -> manual review
input.pipeInput("CHQ001",
"CHQ001|SAV2001|SAV2002|350000|Mumbai");
// Test 2: Normal cheque (75,000) -> auto-cleared
input.pipeInput("CHQ002",
"CHQ002|CUR3001|SAV2003|75000|Delhi");
// Test 3: Invalid cheque (0 amount) -> filtered out
input.pipeInput("CHQ003",
"CHQ003|SAV2004|SAV2005|0|Chennai");
// Test 4: Exactly at threshold (2,00,000) -> auto-cleared
input.pipeInput("CHQ004",
"CHQ004|SAV2006|CUR3002|200000|Kolkata");
System.out.println("Manual review queue: " + manualReview.getQueueSize());
System.out.println(" " + manualReview.readValue());
System.out.println("Auto-cleared queue: " + autoCleared.getQueueSize());
System.out.println(" " + autoCleared.readValue());
System.out.println(" " + autoCleared.readValue());
System.out.println();
System.out.println("CHQ003 (zero amount): filtered out, not in any output");
System.out.println("Test passed: 1 manual review, 2 auto-cleared, 1 filtered");
}
}
}
Expected output:
Manual review queue: 1
CHQ001|SAV2001|SAV2002|350000|Mumbai|CHARGE:350.00
Auto-cleared queue: 2
CHQ002|CUR3001|SAV2003|75000|Delhi|CHARGE:75.00
CHQ004|SAV2006|CUR3002|200000|Kolkata|CHARGE:200.00
CHQ003 (zero amount): filtered out, not in any output
Test passed: 1 manual review, 2 auto-cleared, 1 filtered
How to run
# Step 1: Create topics
kafka-topics.sh --create --topic cheque-events --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic cheque-manual-review --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic cheque-auto-cleared --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
# Step 2: Build
cd cheque-clearing-service
mvn clean package
# Step 3: Run unit test (no broker needed)
java -cp target/cheque-clearing-service-1.0.0.jar com.bank.streams.ChequeClearingTopologyTest
# Step 4: Run the application
java -cp target/cheque-clearing-service-1.0.0.jar com.bank.streams.ChequeClearingApp
# Step 5: Produce test cheques
kafka-console-producer.sh --topic cheque-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type:
# CHQ001:CHQ001|SAV2001|SAV2002|350000|Mumbai
# CHQ002:CHQ002|CUR3001|SAV2003|75000|Delhi
# CHQ003:CHQ003|SAV2004|SAV2005|0|Chennai
# CHQ004:CHQ004|SAV2006|CUR3002|200000|Kolkata
# Step 6: Consume manual review output
kafka-console-consumer.sh --topic cheque-manual-review --from-beginning --bootstrap-server localhost:9092
# Step 7: Consume auto-cleared output
kafka-console-consumer.sh --topic cheque-auto-cleared --from-beginning --bootstrap-server localhost:9092
Expected output on cheque-manual-review:
CHQ001|SAV2001|SAV2002|350000|Mumbai|CHARGE:350.00
Expected output on cheque-auto-cleared:
CHQ002|CUR3001|SAV2003|75000|Delhi|CHARGE:75.00
CHQ004|SAV2006|CUR3002|200000|Kolkata|CHARGE:200.00
CHQ003 does not appear anywhere because the validation filter removed it.
What are common mistakes with topology nodes?
Wrong: forgetting the sink node
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
public class WrongNoSinkNode {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Source and processors, but no sink
builder.stream("bank-transactions")
.filter((key, value) -> value.contains("SUCCESS"))
.mapValues(String::toUpperCase);
// Missing: .to("output-topic")
Topology topology = builder.build();
System.out.println("WRONG: No sink node in topology");
System.out.println();
System.out.println(" What happens:");
System.out.println(" - Topology compiles without error");
System.out.println(" - Application starts without error");
System.out.println(" - Records are read and processed");
System.out.println(" - But results go nowhere (silently lost)");
System.out.println();
System.out.println(" Fix: always end a processing path with .to() or .foreach()");
}
}
Expected output:
WRONG: No sink node in topology
What happens:
- Topology compiles without error
- Application starts without error
- Records are read and processed
- But results go nowhere (silently lost)
Fix: always end a processing path with .to() or .foreach()
Correct: every processing path ends with a sink
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
public class CorrectWithSinkNode {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
builder.stream("bank-transactions")
.filter((key, value) -> value.contains("SUCCESS"))
.mapValues(String::toUpperCase)
.to("successful-transactions"); // Sink node: results written
Topology topology = builder.build();
System.out.println("CORRECT: Processing path ends with .to()");
System.out.println(topology.describe());
}
}
Expected output:
CORRECT: Processing path ends with .to()
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-MAPVALUES-0000000002
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
--> KSTREAM-SINK-0000000003
<-- KSTREAM-FILTER-0000000001
Sink: KSTREAM-SINK-0000000003 (topic: successful-transactions)
<-- KSTREAM-MAPVALUES-0000000002
What should a senior or principal engineer know?
At the senior level, you should be able to read a topology description and trace how data flows from source through processors to sinks. You should also know when to use the Processor API for custom node behavior.
At the principal level, consider:
- Named nodes: use
Named.as("my-filter")to give nodes meaningful names instead of auto-generated ones like KSTREAM-FILTER-0000000001. This makes topology descriptions readable and ensures stability across code changes, which is critical for state store recovery. - Repartition nodes: when you change the key (map, selectKey, groupBy), Kafka Streams inserts an internal repartition topic and additional source/sink nodes. This is expensive. Avoid unnecessary key changes.
- Through nodes: the deprecated
through()method created an explicit intermediate topic. In modern Kafka Streams, userepartition()for explicit repartitioning control. - Topology testing: always call
topology.describe()in tests and compare the output to a known-good baseline. If the describe() output changes unexpectedly, your state store recovery may break. - Fan-out cost: when a source node fans out to multiple processor branches, each branch processes every record independently. If the branches are expensive, consider filtering early to reduce work.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.Named;
public class NamedNodesDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> txns = builder.stream("bank-transactions");
txns
.filter((k, v) -> v.contains("SUCCESS"),
Named.as("filter-successful"))
.mapValues(String::toUpperCase,
Named.as("normalize-uppercase"))
.to("processed-transactions");
Topology topology = builder.build();
System.out.println("Named nodes (readable topology):");
System.out.println(topology.describe());
}
}
Expected output:
Named nodes (readable topology):
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> filter-successful
Processor: filter-successful (stores: [])
--> normalize-uppercase
<-- KSTREAM-SOURCE-0000000000
Processor: normalize-uppercase (stores: [])
--> KSTREAM-SINK-0000000001
<-- filter-successful
Sink: KSTREAM-SINK-0000000001 (topic: processed-transactions)
<-- normalize-uppercase
With Named.as(), the processor nodes show meaningful names instead of auto-generated ones. This makes debugging and monitoring much easier in production.
What are common interview follow-up questions?
- Can a source node read from multiple topics? (Yes. Pass a list of topic names to
builder.stream(Arrays.asList("topic1", "topic2")). All records from both topics flow through the same downstream processors.) - Can a processor node write to multiple downstream nodes? (Yes. Call multiple operations on the same stream. For example, call both
.filter().to("topicA")and.mapValues().to("topicB")on the same source stream.) - What is a state store node? (It is a special processor node that maintains a local key-value store for stateful operations like aggregate, reduce, count, and join. It is backed by a changelog topic for fault tolerance.)
- Can you have a topology with only a source and a sink, no processor? (Yes.
builder.stream("input").to("output")creates a copy topology: source reads and sink writes, with no transformation in between.) - How do repartition nodes work? (When you change the key with map() or selectKey(), Kafka Streams inserts an internal sink node that writes to a repartition topic, and a new source node that reads from it. This ensures records with the same new key end up in the same partition.)
- What happens if a processor node throws an exception? (By default, the stream thread dies. Configure
streams.setUncaughtExceptionHandler()to replace the thread, shut down the client, or shut down the entire application.)
Visual Overview
Node Types in a Topology
=========================
┌──────────────────────┐
│ SOURCE NODE │
│ Reads from Kafka │
│ topic(s) │
│ builder.stream() │
└──────────┬───────────┘
│ records
▼
┌──────────────────────┐
│ PROCESSOR NODE 1 │
│ filter() │
│ Remove invalid │
└──────────┬───────────┘
│ valid records
▼
┌──────────────────────┐
│ PROCESSOR NODE 2 │
│ mapValues() │
│ Calculate charges │
└──────┬──────┬────────┘
│ │
│ │ branch
▼ ▼
┌──────────┐ ┌──────────┐
│ SINK 1 │ │ SINK 2 │
│ manual │ │ auto │
│ review │ │ cleared │
│ .to() │ │ .to() │
└──────────┘ └──────────┘
Cheque Clearing Pipeline
=========================
cheque-events (topic)
│
▼ Source
┌─────────────┐
│ Validate │ filter: amount > 0
└──────┬──────┘
▼ Processor
┌─────────────┐
│ Charge calc │ mapValues: 0.1% charge
└──────┬──────┘
│
┌────┴────┐
▼ ▼
┌─────┐ ┌─────┐
│>200K│ │<=200K│
│Sink │ │Sink │
└──┬──┘ └──┬──┘
▼ ▼
manual auto
review cleared
Key Takeaway
Every Kafka Streams topology is built from three types of nodes. Source nodes read records from Kafka topics and push them into the processing graph. Processor nodes receive records, apply transformation logic like filtering, mapping, joining, or aggregating, and forward the results downstream. Sink nodes receive the final results and write them to Kafka output topics.
In the DSL, these nodes are created implicitly: builder.stream() creates a source, each chained operation like filter() or mapValues() creates a processor, and .to() creates a sink. In the Processor API, you add them explicitly with addSource(), addProcessor(), and addSink(). Either way, the result is a directed graph where data flows from left to right.
In our National Bank of India cheque clearing project, one source node read cheque events, one processor validated amounts, another calculated clearing charges, two more processors branched by amount threshold, and two sink nodes routed high-value cheques to manual review and normal cheques to auto-clearing. The unit test verified all four cases: high-value, normal, zero amount (filtered out), and exactly at threshold.
At the principal level, use Named.as() to give nodes meaningful names for stable topology descriptions, watch for repartition nodes that appear when you change keys, and always end every processing path with a sink node so results are not silently lost.
Remember one sentence: a topology is a graph of source nodes that read, processor nodes that transform, and sink nodes that write, and every processing path must start at a source and end at a sink.
---