Definition
- StreamsBuilder is the entry point for the Kafka Streams DSL: you call methods like
stream(),filter(),mapValues(),groupByKey(), andto()on it to define your processing pipeline. - Topology is the immutable, compiled processing graph produced by
builder.build(): it contains source nodes (read from topics), processor nodes (transformations), and sink nodes (write to topics). - KafkaStreams is the runtime class that takes a Topology and a Properties config, creates stream threads, assigns partitions, manages state stores, commits offsets, and runs the processing loop until you call
close(). - Lifecycle order is always: StreamsBuilder (define), Topology (compile), KafkaStreams (run).
- Processor API is the low-level alternative to StreamsBuilder: you add source, processor, and sink nodes manually to a Topology object, giving you full control but more code.
Think of building a loan processing pipeline at National Bank of India. StreamsBuilder is the architect who draws the blueprint: receive application, verify documents, calculate EMI, send approval. The build() call prints the final blueprint as a Topology. KafkaStreams is the construction team that takes that blueprint, hires workers (threads), assigns them to branch offices (partitions), and runs the actual loan processing every day. The architect defines the plan, the blueprint freezes it, and the team executes it.
Understanding the Concepts
What is StreamsBuilder?
StreamsBuilder is the class you use to define your Kafka Streams processing logic. It provides the DSL (Domain Specific Language) methods that let you describe what your application should do with the data.
You call stream() to read from a topic as a KStream, table() to read as a KTable, and then chain operations like filter(), mapValues(), groupByKey(), join(), and to().
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import java.math.BigDecimal;
public class StreamsBuilderDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// stream() creates a KStream from a topic
KStream<String, String> transactions = builder.stream("bank-transactions");
// table() creates a KTable from a topic (latest value per key)
KTable<String, String> accountBalances = builder.table("account-balances");
// Chain operations on KStream
transactions
.filter((accountId, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[1]);
return amount.compareTo(new BigDecimal("10000")) > 0;
})
.mapValues(value -> value.toUpperCase())
.to("large-transactions");
System.out.println("StreamsBuilder methods used:");
System.out.println(" builder.stream('bank-transactions') -> KStream");
System.out.println(" builder.table('account-balances') -> KTable");
System.out.println(" .filter() -> keep records matching condition");
System.out.println(" .mapValues() -> transform the value");
System.out.println(" .to() -> write to output topic");
}
}
Expected output:
StreamsBuilder methods used:
builder.stream('bank-transactions') -> KStream
builder.table('account-balances') -> KTable
.filter() -> keep records matching condition
.mapValues() -> transform the value
.to() -> write to output topic
What is Topology?
Topology is the compiled, immutable processing graph. When you call builder.build(), StreamsBuilder takes all the operations you defined and produces a Topology object. This object contains every source, processor, and sink node and how they connect.
You can inspect a Topology by calling describe(). This is useful for debugging, logging, and verifying your processing plan before starting the application.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
public class TopologyDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> loans = builder.stream("loan-applications");
loans
.filter((loanId, value) -> value.contains("APPROVED"))
.mapValues(value -> "PROCESSED: " + value.toUpperCase())
.to("processed-loans");
// build() compiles the DSL into a Topology
Topology topology = builder.build();
// describe() shows every node and connection
System.out.println("Topology description:");
System.out.println(topology.describe());
}
}
Expected output:
Topology description:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [loan-applications])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-MAPVALUES-0000000002
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
--> KSTREAM-SINK-0000000003
<-- KSTREAM-FILTER-0000000001
Sink: KSTREAM-SINK-0000000003 (topic: processed-loans)
<-- KSTREAM-MAPVALUES-0000000002
Each line shows a node, its type (Source, Processor, Sink), the stores it uses, and the nodes it connects to (--> downstream, <-- upstream).
What is KafkaStreams?
KafkaStreams is the runtime engine. It takes a Topology and a Properties configuration, then manages everything needed to run the processing: creating threads, assigning partitions, initializing state stores, polling records, executing the topology, committing offsets, and shutting down gracefully.
You create a KafkaStreams instance, call start() to begin processing, and call close() to stop. A shutdown hook ensures clean shutdown on application termination.
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class KafkaStreamsDemo {
public static void main(String[] args) {
// 1. Configuration
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
// 2. Define processing with StreamsBuilder
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> loans = builder.stream("loan-applications");
loans.filter((k, v) -> v.contains("APPROVED")).to("approved-loans");
// 3. Compile to Topology
Topology topology = builder.build();
// 4. Create KafkaStreams runtime
KafkaStreams streams = new KafkaStreams(topology, props);
CountDownLatch latch = new CountDownLatch(1);
// 5. Shutdown hook for graceful stop
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Application stopped");
}));
// 6. Start processing
try {
streams.start();
System.out.println("KafkaStreams started");
System.out.println(" State: " + streams.state());
System.out.println(" Application ID: loan-processor");
System.out.println(" Input: loan-applications");
System.out.println(" Output: approved-loans");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
Expected output:
KafkaStreams started
State: RUNNING
Application ID: loan-processor
Input: loan-applications
Output: approved-loans
How do StreamsBuilder, Topology, and KafkaStreams work together?
The three classes have a strict lifecycle order. You always follow these steps:
- Create a StreamsBuilder and define your processing logic.
- Call
builder.build()to compile the logic into a Topology. - Create a KafkaStreams instance with the Topology and configuration.
- Call
streams.start()to begin processing. - Call
streams.close()to stop (usually in a shutdown hook).
| Class | Role | When | Methods you call |
|---|---|---|---|
| StreamsBuilder | Define the processing logic | At startup, before build | stream(), table(), filter(), map(), to() |
| Topology | Compiled processing plan | After build(), before start | describe() to inspect |
| KafkaStreams | Runtime engine | After topology is built | start(), close(), state() |
package com.bank.streams;
public class LifecycleOrder {
public static void main(String[] args) {
System.out.println("Kafka Streams lifecycle:");
System.out.println();
System.out.println("Step 1: StreamsBuilder builder = new StreamsBuilder();");
System.out.println(" builder.stream('topic').filter(...).to('output');");
System.out.println(" -> Defines WHAT to do");
System.out.println();
System.out.println("Step 2: Topology topology = builder.build();");
System.out.println(" -> Compiles the plan (immutable after this)");
System.out.println();
System.out.println("Step 3: KafkaStreams streams = new KafkaStreams(topology, props);");
System.out.println(" -> Creates the runtime (not yet running)");
System.out.println();
System.out.println("Step 4: streams.start();");
System.out.println(" -> Connects to Kafka, creates threads, begins processing");
System.out.println();
System.out.println("Step 5: streams.close();");
System.out.println(" -> Commits offsets, closes connections, stops threads");
}
}
Expected output:
Kafka Streams lifecycle:
Step 1: StreamsBuilder builder = new StreamsBuilder();
builder.stream('topic').filter(...).to('output');
-> Defines WHAT to do
Step 2: Topology topology = builder.build();
-> Compiles the plan (immutable after this)
Step 3: KafkaStreams streams = new KafkaStreams(topology, props);
-> Creates the runtime (not yet running)
Step 4: streams.start();
-> Connects to Kafka, creates threads, begins processing
Step 5: streams.close();
-> Commits offsets, closes connections, stops threads
What is the Processor API and how does it differ from StreamsBuilder?
StreamsBuilder is the high-level DSL. The Processor API is the low-level alternative where you add nodes to a Topology directly. The Processor API gives you more control but requires more code.
Most applications use StreamsBuilder. The Processor API is used when you need custom processing logic that the DSL does not support, like accessing the ProcessorContext to schedule punctuations or forward records conditionally.
package com.bank.streams;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.processor.api.Processor;
import org.apache.kafka.streams.processor.api.ProcessorContext;
import org.apache.kafka.streams.processor.api.Record;
import org.apache.kafka.common.serialization.Serdes;
import java.math.BigDecimal;
public class ProcessorApiDemo {
// Custom processor that filters high-value transactions
public static class HighValueFilter
implements Processor<String, String, String, String> {
private ProcessorContext<String, String> context;
private static final BigDecimal THRESHOLD = new BigDecimal("100000");
@Override
public void init(ProcessorContext<String, String> context) {
this.context = context;
}
@Override
public void process(Record<String, String> record) {
String[] parts = record.value().split("\\|");
BigDecimal amount = new BigDecimal(parts[1]);
if (amount.compareTo(THRESHOLD) > 0) {
context.forward(record);
}
}
@Override
public void close() {}
}
public static void main(String[] args) {
Topology topology = new Topology();
// Manually add source, processor, and sink nodes
topology.addSource("Source", "bank-transactions");
topology.addProcessor("Filter", HighValueFilter::new, "Source");
topology.addSink("Sink", "high-value-alerts",
Serdes.String().serializer(),
Serdes.String().serializer(),
"Filter");
System.out.println("Processor API topology:");
System.out.println(topology.describe());
System.out.println();
System.out.println("DSL (StreamsBuilder): builder.stream().filter().to()");
System.out.println("Processor API: topology.addSource().addProcessor().addSink()");
System.out.println();
System.out.println("Use DSL for most cases. Use Processor API when you need:");
System.out.println(" - Custom punctuation (scheduled callbacks)");
System.out.println(" - Conditional forwarding to different downstream nodes");
System.out.println(" - Direct access to state stores in custom logic");
System.out.println(" - Headers manipulation");
}
}
Expected output:
Processor API topology:
Topologies:
Sub-topology: 0
Source: Source (topics: [bank-transactions])
--> Filter
Processor: Filter (stores: [])
--> Sink
<-- Source
Sink: Sink (topic: high-value-alerts)
<-- Filter
DSL (StreamsBuilder): builder.stream().filter().to()
Processor API: topology.addSource().addProcessor().addSink()
Use DSL for most cases. Use Processor API when you need:
- Custom punctuation (scheduled callbacks)
- Conditional forwarding to different downstream nodes
- Direct access to state stores in custom logic
- Headers manipulation
What is a complete banking project using all three classes?
This project builds an EMI (Equated Monthly Installment) calculator service. It reads approved loan applications, calculates the monthly EMI, and writes the result. The project shows StreamsBuilder, Topology, and KafkaStreams working together.
Project structure
emi-calculator-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── EmiCalculatorApp.java (main: KafkaStreams)
│ ├── EmiCalculatorTopology.java (StreamsBuilder + Topology)
│ └── LoanApplication.java (model)
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── EmiCalculatorTopologyTest.java (TopologyTestDriver)
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>emi-calculator-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
LoanApplication.java
package com.bank.streams;
import java.math.BigDecimal;
public class LoanApplication {
private String loanId;
private String customerId;
private BigDecimal principal;
private BigDecimal annualRate;
private int tenureMonths;
private String status;
public LoanApplication() {}
public LoanApplication(String loanId, String customerId, BigDecimal principal,
BigDecimal annualRate, int tenureMonths, String status) {
this.loanId = loanId;
this.customerId = customerId;
this.principal = principal;
this.annualRate = annualRate;
this.tenureMonths = tenureMonths;
this.status = status;
}
public String getLoanId() { return loanId; }
public String getCustomerId() { return customerId; }
public BigDecimal getPrincipal() { return principal; }
public BigDecimal getAnnualRate() { return annualRate; }
public int getTenureMonths() { return tenureMonths; }
public String getStatus() { return status; }
@Override
public String toString() {
return loanId + "|" + customerId + "|" + principal + "|" + annualRate
+ "|" + tenureMonths + "|" + status;
}
public static LoanApplication fromString(String value) {
String[] p = value.split("\\|");
return new LoanApplication(
p[0], p[1], new BigDecimal(p[2]),
new BigDecimal(p[3]), Integer.parseInt(p[4]), p[5]);
}
}
EmiCalculatorTopology.java (StreamsBuilder + Topology)
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
import java.math.MathContext;
import java.math.RoundingMode;
public class EmiCalculatorTopology {
/**
* EMI formula: EMI = P * r * (1+r)^n / ((1+r)^n - 1)
* P = principal, r = monthly rate, n = tenure in months
*/
public static BigDecimal calculateEmi(BigDecimal principal,
BigDecimal annualRate,
int tenureMonths) {
BigDecimal monthlyRate = annualRate
.divide(new BigDecimal("12"), 10, RoundingMode.HALF_UP)
.divide(new BigDecimal("100"), 10, RoundingMode.HALF_UP);
// (1 + r)^n
BigDecimal onePlusR = BigDecimal.ONE.add(monthlyRate);
BigDecimal power = onePlusR.pow(tenureMonths, MathContext.DECIMAL128);
// EMI = P * r * (1+r)^n / ((1+r)^n - 1)
BigDecimal numerator = principal.multiply(monthlyRate).multiply(power);
BigDecimal denominator = power.subtract(BigDecimal.ONE);
return numerator.divide(denominator, 2, RoundingMode.HALF_UP);
}
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> applications = builder.stream("loan-applications");
// Step 1: Filter only approved loans
KStream<String, String> approved = applications
.filter((loanId, value) -> {
LoanApplication app = LoanApplication.fromString(value);
return "APPROVED".equals(app.getStatus());
});
// Step 2: Calculate EMI and append to the record
KStream<String, String> withEmi = approved
.mapValues(value -> {
LoanApplication app = LoanApplication.fromString(value);
BigDecimal emi = calculateEmi(
app.getPrincipal(), app.getAnnualRate(), app.getTenureMonths());
return app.toString() + "|EMI:" + emi;
});
// Step 3: Write to output topic
withEmi.to("loan-emi-results");
return builder.build();
}
}
EmiCalculatorApp.java (KafkaStreams runtime)
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class EmiCalculatorApp {
public static void main(String[] args) {
// Configuration
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "emi-calculator-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
// Build Topology from StreamsBuilder
Topology topology = EmiCalculatorTopology.build();
System.out.println("Topology:");
System.out.println(topology.describe());
// Create and start KafkaStreams runtime
KafkaStreams streams = new KafkaStreams(topology, props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("EMI calculator stopped");
}));
try {
streams.start();
System.out.println("EMI calculator started");
System.out.println(" Application ID: emi-calculator-service");
System.out.println(" Threads: 2");
System.out.println(" Input topic: loan-applications");
System.out.println(" Output topic: loan-emi-results");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
EmiCalculatorTopologyTest.java (unit test)
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class EmiCalculatorTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(EmiCalculatorTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"loan-applications",
new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"loan-emi-results",
new StringDeserializer(), new StringDeserializer());
// Test 1: Approved loan, 5,00,000 at 8.5% for 60 months
// Expected EMI: 10,242.38
input.pipeInput("LOAN001",
"LOAN001|CUST101|500000|8.5|60|APPROVED");
// Test 2: Rejected loan (should be filtered out)
input.pipeInput("LOAN002",
"LOAN002|CUST102|300000|9.0|36|REJECTED");
// Test 3: Approved loan, 10,00,000 at 7.5% for 120 months
// Expected EMI: 11,870.27
input.pipeInput("LOAN003",
"LOAN003|CUST103|1000000|7.5|120|APPROVED");
System.out.println("Output records: " + output.getQueueSize());
System.out.println("Record 1: " + output.readValue());
System.out.println("Record 2: " + output.readValue());
System.out.println("Queue empty: " + output.isEmpty());
System.out.println();
System.out.println("Test passed: 2 approved loans processed, 1 rejected filtered out");
}
}
}
Expected output:
Output records: 2
Record 1: LOAN001|CUST101|500000|8.5|60|APPROVED|EMI:10242.38
Record 2: LOAN003|CUST103|1000000|7.5|120|APPROVED|EMI:11870.27
Queue empty: true
Test passed: 2 approved loans processed, 1 rejected filtered out
How to run
# Step 1: Create topics
kafka-topics.sh --create --topic loan-applications --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic loan-emi-results --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
# Step 2: Build
cd emi-calculator-service
mvn clean package
# Step 3: Run the unit test first (no broker needed)
java -cp target/emi-calculator-service-1.0.0.jar com.bank.streams.EmiCalculatorTopologyTest
# Step 4: Run the application
java -cp target/emi-calculator-service-1.0.0.jar com.bank.streams.EmiCalculatorApp
# Step 5: Produce test loan applications
kafka-console-producer.sh --topic loan-applications --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type:
# LOAN001:LOAN001|CUST101|500000|8.5|60|APPROVED
# LOAN002:LOAN002|CUST102|300000|9.0|36|REJECTED
# LOAN003:LOAN003|CUST103|1000000|7.5|120|APPROVED
# Step 6: Consume output
kafka-console-consumer.sh --topic loan-emi-results --from-beginning --bootstrap-server localhost:9092
Expected output on loan-emi-results consumer:
LOAN001|CUST101|500000|8.5|60|APPROVED|EMI:10242.38
LOAN003|CUST103|1000000|7.5|120|APPROVED|EMI:11870.27
LOAN002 (REJECTED) does not appear because the filter removed it.
What are common mistakes with StreamsBuilder, Topology, and KafkaStreams?
Wrong: calling build() multiple times
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
public class WrongMultipleBuild {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
builder.stream("transactions").to("output-1");
Topology t1 = builder.build();
// WRONG: adding more operations after build()
// Some versions of Kafka Streams allow this, but it leads to confusion
// because t1 does not include operations added after build()
builder.stream("accounts").to("output-2");
Topology t2 = builder.build();
System.out.println("WRONG: Do not call build() multiple times");
System.out.println();
System.out.println(" Problem: behavior varies across Kafka versions");
System.out.println(" Some versions: t2 includes both, t1 does not");
System.out.println(" Other versions: second build() throws exception");
System.out.println();
System.out.println(" Rule: define all operations first, call build() once");
}
}
Expected output:
WRONG: Do not call build() multiple times
Problem: behavior varies across Kafka versions
Some versions: t2 includes both, t1 does not
Other versions: second build() throws exception
Rule: define all operations first, call build() once
Correct: define everything, then build once
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
public class CorrectSingleBuild {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Define ALL operations first
builder.stream("transactions")
.filter((k, v) -> v.contains("SUCCESS"))
.to("successful-transactions");
builder.stream("accounts")
.filter((k, v) -> v.contains("ACTIVE"))
.to("active-accounts");
// Build once
Topology topology = builder.build();
System.out.println("CORRECT: All operations defined before build()");
System.out.println(topology.describe());
}
}
Expected output:
CORRECT: All operations defined before build()
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [transactions])
--> KSTREAM-FILTER-0000000001
...
Sub-topology: 1
Source: KSTREAM-SOURCE-0000000003 (topics: [accounts])
--> KSTREAM-FILTER-0000000004
...
What should a senior or principal engineer know?
At the senior level, you should understand the lifecycle completely and know when to use the DSL versus the Processor API.
At the principal level, consider:
- Topology immutability: once
build()is called, the Topology cannot be changed. This means you cannot dynamically add or remove processing steps at runtime. If the processing logic needs to change, you must restart the application with a new Topology. - State listener: register a
KafkaStreams.StateListenerto monitor state transitions (CREATED, REBALANCING, RUNNING, PENDING_SHUTDOWN, NOT_RUNNING, ERROR). Alert on ERROR state. - Uncaught exception handler: set
streams.setUncaughtExceptionHandler()to control what happens when a stream thread throws an unexpected exception. Options are REPLACE_THREAD (restart the thread), SHUTDOWN_CLIENT (stop this instance), or SHUTDOWN_APPLICATION (stop all instances). - Named topologies (Kafka 3.x): allow running multiple independent topologies in a single KafkaStreams instance, each with its own set of input topics and application semantics.
- Topology compatibility: when upgrading, ensure the new Topology is compatible with the old one. Changing the order of operations or renaming internal topics can break state store recovery.
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.errors.StreamsUncaughtExceptionHandler;
public class PrincipalLifecycleConfig {
public static void main(String[] args) {
System.out.println("Principal-level lifecycle management:");
System.out.println();
System.out.println("1. State listener:");
System.out.println(" streams.setStateListener((newState, oldState) ->");
System.out.println(" log.info('State: ' + oldState + ' -> ' + newState));");
System.out.println(" Monitor: CREATED -> REBALANCING -> RUNNING");
System.out.println(" Alert on: ERROR state");
System.out.println();
System.out.println("2. Uncaught exception handler:");
System.out.println(" REPLACE_THREAD: restart failed thread");
System.out.println(" SHUTDOWN_CLIENT: stop this instance");
System.out.println(" SHUTDOWN_APPLICATION: stop all instances");
System.out.println();
System.out.println("3. Topology compatibility on upgrade:");
System.out.println(" - Do not reorder operations");
System.out.println(" - Do not rename internal repartition topics");
System.out.println(" - Use named processors: .filter(Named.as('myFilter'), ...)");
System.out.println(" - Test: build old and new Topology, compare describe() output");
}
}
Expected output:
Principal-level lifecycle management:
1. State listener:
streams.setStateListener((newState, oldState) ->
log.info('State: ' + oldState + ' -> ' + newState));
Monitor: CREATED -> REBALANCING -> RUNNING
Alert on: ERROR state
2. Uncaught exception handler:
REPLACE_THREAD: restart failed thread
SHUTDOWN_CLIENT: stop this instance
SHUTDOWN_APPLICATION: stop all instances
3. Topology compatibility on upgrade:
- Do not reorder operations
- Do not rename internal repartition topics
- Use named processors: .filter(Named.as('myFilter'), ...)
- Test: build old and new Topology, compare describe() output
What are common interview follow-up questions?
- Can you modify a Topology after calling build()? (No. Topology is immutable after build(). You must create a new StreamsBuilder and build a new Topology.)
- What is the difference between StreamsBuilder and Topology? (StreamsBuilder is the DSL for defining logic. Topology is the compiled graph. StreamsBuilder uses Topology internally.)
- Can you reuse a KafkaStreams instance after calling close()? (No. Once closed, you must create a new KafkaStreams instance. You cannot call start() again.)
- What are the KafkaStreams states? (CREATED, REBALANCING, RUNNING, PENDING_SHUTDOWN, NOT_RUNNING, ERROR. Normal flow is CREATED -> REBALANCING -> RUNNING.)
- Should you put the Topology definition in a separate class? (Yes. Separating the Topology from the main class makes it testable with TopologyTestDriver without starting the full application.)
- What happens if you do not add a shutdown hook? (The application may not commit final offsets or close state stores cleanly, leading to duplicate processing on restart.)
Visual Overview
StreamsBuilder, Topology, KafkaStreams Lifecycle
==================================================
┌────────────────────────────────┐
│ StreamsBuilder │
│ │
│ stream('loan-applications') │
│ .filter(status == APPROVED) │
│ .mapValues(calculate EMI) │
│ .to('loan-emi-results') │
│ │
│ DEFINE the processing logic │
└───────────────┬────────────────┘
│ build()
▼
┌────────────────────────────────┐
│ Topology │
│ │
│ Source ──> Filter ──> MapValues│
│ ──> Sink │
│ │
│ COMPILED processing plan │
│ (immutable) │
└───────────────┬────────────────┘
│ new KafkaStreams(topology, props)
▼
┌────────────────────────────────┐
│ KafkaStreams │
│ │
│ start() -> RUNNING │
│ state() -> monitor │
│ close() -> shutdown │
│ │
│ RUNTIME engine │
│ (threads, partitions, offsets) │
└────────────────────────────────┘
DSL vs Processor API
=====================
DSL (StreamsBuilder): High-level, declarative
builder.stream().filter().mapValues().to()
Processor API (Topology): Low-level, imperative
topology.addSource().addProcessor().addSink()
Key Takeaway
StreamsBuilder, Topology, and KafkaStreams are the three core classes in every Kafka Streams application. StreamsBuilder is where you define what your application does: read from topics, filter, transform, join, aggregate, and write to topics. The build() method compiles those definitions into a Topology, which is an immutable directed graph of processing nodes.
KafkaStreams is the runtime that executes the Topology. It creates threads, assigns partitions, initializes state stores, polls records from Kafka, runs them through the topology, and commits offsets. You call start() to begin and close() to stop, and a shutdown hook ensures clean shutdown.
In our National Bank of India EMI calculator project, we defined the processing in EmiCalculatorTopology using StreamsBuilder: read loan applications, filter approved ones, calculate the monthly EMI using the standard formula, and write results. We compiled it with build(), then EmiCalculatorApp created a KafkaStreams instance and started it. The unit test used TopologyTestDriver to verify the EMI calculation without a running broker.
The Processor API is the low-level alternative to StreamsBuilder. You add source, processor, and sink nodes to a Topology directly. Use it only when you need custom punctuation, conditional forwarding, or direct state store access. For everything else, the DSL is simpler and less error-prone.
Remember one sentence: StreamsBuilder defines the logic, build() compiles it into an immutable Topology, and KafkaStreams executes that Topology as a fault-tolerant, scalable processing engine.
---