Definition
- Topology is a directed acyclic graph of processing nodes: source nodes read from Kafka topics, processor nodes transform data, and sink nodes write to Kafka topics.
- Stream task is the smallest unit of parallelism: Kafka Streams creates one task per input partition, and each task processes records from that partition through its own copy of the topology.
- Stream thread is a Java thread that executes one or more stream tasks; you configure the count with
num.stream.threads. - Sub-topology is an independent section of the full topology that does not share data with other sections; Kafka Streams can assign sub-topology tasks to different threads.
- Application instance is one running JVM process; multiple instances with the same
application.idform a group and divide tasks among themselves, just like Kafka consumer groups.
Think of National Bank of India processing loan applications. The topology is the process chart on the wall: receive application (source), verify documents (processor), calculate interest (processor), send approval letter (sink). Each branch office (stream task) has its own copy of that chart and handles applications from its assigned district (partition). Each loan officer (stream thread) works in one or more branches. The bank can open more branches (more tasks) only if there are more districts, and it can hire more officers (more threads) to handle branches faster. Opening a second building (another JVM instance) lets the bank spread branches across buildings.
Understanding the Concepts
What is a topology?
A topology is the complete processing plan for a Kafka Streams application. It is a directed graph with three types of nodes.
A source node reads records from one or more Kafka topics. A processor node receives records from upstream nodes, applies logic (filter, map, join, aggregate), and forwards results to downstream nodes. A sink node writes records to a Kafka topic.
You build a topology using the StreamsBuilder DSL. Each method call adds a node to the graph.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
public class LoanTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// Source node: read loan applications
KStream<String, String> applications = builder.stream("loan-applications");
// Processor node 1: filter approved applications
KStream<String, String> approved = applications
.filter((loanId, value) -> value.contains("APPROVED"));
// Processor node 2: calculate interest and tag
KStream<String, String> withInterest = approved
.mapValues(value -> {
String[] parts = value.split("\\|");
BigDecimal principal = new BigDecimal(parts[1]);
BigDecimal interest = principal
.multiply(new BigDecimal("0.085"))
.setScale(2, java.math.RoundingMode.HALF_UP);
return value + "|INTEREST:" + interest;
});
// Sink node: write to processed topic
withInterest.to("processed-loans");
Topology topology = builder.build();
System.out.println(topology.describe());
return topology;
}
public static void main(String[] args) {
build();
}
}
Expected output:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [loan-applications])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-MAPVALUES-0000000002
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
--> KSTREAM-SINK-0000000003
<-- KSTREAM-FILTER-0000000001
Sink: KSTREAM-SINK-0000000003 (topic: processed-loans)
<-- KSTREAM-MAPVALUES-0000000002
The output shows one sub-topology with four nodes: source, filter, mapValues, and sink. Each arrow shows the data flow direction.
What is a stream task?
A stream task is the unit of parallelism in Kafka Streams. Kafka Streams creates one task per input partition. Each task runs its own independent copy of the topology and processes records from its assigned partition.
If the loan-applications topic has 6 partitions, Kafka Streams creates 6 tasks. Each task reads from exactly one partition and processes all records from that partition in order.
package com.bank.streams;
public class StreamTaskDemo {
public static void main(String[] args) {
System.out.println("Topic: loan-applications (6 partitions)");
System.out.println();
System.out.println("Kafka Streams creates 6 tasks:");
System.out.println(" Task 0_0: reads partition 0");
System.out.println(" Task 0_1: reads partition 1");
System.out.println(" Task 0_2: reads partition 2");
System.out.println(" Task 0_3: reads partition 3");
System.out.println(" Task 0_4: reads partition 4");
System.out.println(" Task 0_5: reads partition 5");
System.out.println();
System.out.println("Task naming: <sub-topology-id>_<partition-number>");
System.out.println(" 0_0 means sub-topology 0, partition 0");
System.out.println();
System.out.println("Each task:");
System.out.println(" - Has its own copy of the topology");
System.out.println(" - Processes records from one partition only");
System.out.println(" - Maintains its own state store (if stateful)");
System.out.println(" - Commits offsets independently");
}
}
Expected output:
Topic: loan-applications (6 partitions)
Kafka Streams creates 6 tasks:
Task 0_0: reads partition 0
Task 0_1: reads partition 1
Task 0_2: reads partition 2
Task 0_3: reads partition 3
Task 0_4: reads partition 4
Task 0_5: reads partition 5
Task naming: <sub-topology-id>_<partition-number>
0_0 means sub-topology 0, partition 0
Each task:
- Has its own copy of the topology
- Processes records from one partition only
- Maintains its own state store (if stateful)
- Commits offsets independently
What is a stream thread?
A stream thread is a Java thread inside your application that executes one or more stream tasks. You configure the number of threads with the num.stream.threads property.
If you have 6 tasks and 2 threads, each thread handles 3 tasks. If you have 6 tasks and 6 threads, each thread handles 1 task. Having more threads than tasks wastes resources because the extra threads sit idle.
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;
public class StreamThreadConfig {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 3);
int partitions = 6;
int threads = 3;
int tasksPerThread = partitions / threads;
System.out.println("Configuration:");
System.out.println(" Partitions: " + partitions);
System.out.println(" Stream threads: " + threads);
System.out.println(" Tasks per thread: " + tasksPerThread);
System.out.println();
System.out.println("Thread assignment:");
System.out.println(" Thread-1: Task 0_0, Task 0_1");
System.out.println(" Thread-2: Task 0_2, Task 0_3");
System.out.println(" Thread-3: Task 0_4, Task 0_5");
System.out.println();
System.out.println("Rule: num.stream.threads <= partition count");
System.out.println(" More threads than tasks = idle threads");
System.out.println(" Fewer threads than tasks = threads handle multiple tasks");
}
}
Expected output:
Configuration:
Partitions: 6
Stream threads: 3
Tasks per thread: 2
Thread assignment:
Thread-1: Task 0_0, Task 0_1
Thread-2: Task 0_2, Task 0_3
Thread-3: Task 0_4, Task 0_5
Rule: num.stream.threads <= partition count
More threads than tasks = idle threads
Fewer threads than tasks = threads handle multiple tasks
How do partitions drive parallelism?
The number of input partitions is the hard limit on parallelism. You cannot process records faster by adding more threads or instances than there are partitions.
| Partitions | Threads (1 instance) | Instances | Total threads | Active tasks | Idle threads |
|---|---|---|---|---|---|
| 6 | 2 | 1 | 2 | 6 (2 threads x 3 tasks each) | 0 |
| 6 | 6 | 1 | 6 | 6 (1 task per thread) | 0 |
| 6 | 8 | 1 | 8 | 6 | 2 idle |
| 6 | 2 | 3 | 6 | 6 (2 tasks per instance) | 0 |
| 6 | 3 | 3 | 9 | 6 | 3 idle |
| 12 | 4 | 3 | 12 | 12 (4 tasks per instance) | 0 |
package com.bank.streams;
public class ParallelismDemo {
public static void main(String[] args) {
int partitions = 6;
System.out.println("Topic: loan-applications (" + partitions + " partitions)");
System.out.println();
// Scenario 1: 1 instance, 2 threads
System.out.println("Scenario 1: 1 instance, 2 threads");
System.out.println(" Thread-1 handles tasks: 0_0, 0_1, 0_2");
System.out.println(" Thread-2 handles tasks: 0_3, 0_4, 0_5");
System.out.println(" All 6 partitions covered");
System.out.println();
// Scenario 2: 3 instances, 2 threads each
System.out.println("Scenario 2: 3 instances, 2 threads each");
System.out.println(" Instance A, Thread-1: task 0_0");
System.out.println(" Instance A, Thread-2: task 0_1");
System.out.println(" Instance B, Thread-1: task 0_2");
System.out.println(" Instance B, Thread-2: task 0_3");
System.out.println(" Instance C, Thread-1: task 0_4");
System.out.println(" Instance C, Thread-2: task 0_5");
System.out.println(" All 6 partitions covered, spread across 3 JVMs");
System.out.println();
// Scenario 3: too many threads
System.out.println("Scenario 3: 1 instance, 8 threads (too many)");
System.out.println(" Threads 1-6 each get 1 task");
System.out.println(" Threads 7-8 sit idle, wasting memory");
System.out.println();
System.out.println("Conclusion: max useful parallelism = partition count");
}
}
Expected output:
Topic: loan-applications (6 partitions)
Scenario 1: 1 instance, 2 threads
Thread-1 handles tasks: 0_0, 0_1, 0_2
Thread-2 handles tasks: 0_3, 0_4, 0_5
All 6 partitions covered
Scenario 2: 3 instances, 2 threads each
Instance A, Thread-1: task 0_0
Instance A, Thread-2: task 0_1
Instance B, Thread-1: task 0_2
Instance B, Thread-2: task 0_3
Instance C, Thread-1: task 0_4
Instance C, Thread-2: task 0_5
All 6 partitions covered, spread across 3 JVMs
Scenario 3: 1 instance, 8 threads (too many)
Threads 1-6 each get 1 task
Threads 7-8 sit idle, wasting memory
Conclusion: max useful parallelism = partition count
What are sub-topologies and why do they matter?
A full topology may contain independent sections. If you read from two unrelated topics and process them separately, Kafka Streams splits them into separate sub-topologies. Each sub-topology creates its own set of tasks.
This matters because tasks from different sub-topologies can run on different threads or instances, reducing unnecessary data shuffling.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
public class SubTopologyDemo {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// Sub-topology 0: loan processing
KStream<String, String> loans = builder.stream("loan-applications");
loans.filter((k, v) -> v.contains("APPROVED")).to("approved-loans");
// Sub-topology 1: account alerts (independent, no shared data)
KStream<String, String> accounts = builder.stream("account-updates");
accounts.filter((k, v) -> v.contains("CLOSED")).to("closed-account-alerts");
Topology topology = builder.build();
System.out.println(topology.describe());
return topology;
}
public static void main(String[] args) {
build();
}
}
Expected output:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [loan-applications])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-SINK-0000000002
<-- KSTREAM-SOURCE-0000000000
Sink: KSTREAM-SINK-0000000002 (topic: approved-loans)
<-- KSTREAM-FILTER-0000000001
Sub-topology: 1
Source: KSTREAM-SOURCE-0000000003 (topics: [account-updates])
--> KSTREAM-FILTER-0000000004
Processor: KSTREAM-FILTER-0000000004 (stores: [])
--> KSTREAM-SINK-0000000005
<-- KSTREAM-SOURCE-0000000003
Sink: KSTREAM-SINK-0000000005 (topic: closed-account-alerts)
<-- KSTREAM-FILTER-0000000004
Kafka Streams created two separate sub-topologies. If loan-applications has 4 partitions and account-updates has 3 partitions, there will be 4 + 3 = 7 tasks total.
What is the complete banking project for understanding architecture?
This project demonstrates topology, tasks, threads, and scaling. It processes loan applications, logs which thread and task handle each record, and shows how adding threads changes the distribution.
Project structure
loan-processor/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── LoanProcessorApp.java
├── LoanProcessorTopology.java
└── LoanApplication.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>loan-processorsyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
LoanApplication.java
package com.bank.streams;
import java.math.BigDecimal;
public class LoanApplication {
private String loanId;
private String customerId;
private BigDecimal amount;
private String status;
private String branch;
public LoanApplication() {}
public LoanApplication(String loanId, String customerId, BigDecimal amount,
String status, String branch) {
this.loanId = loanId;
this.customerId = customerId;
this.amount = amount;
this.status = status;
this.branch = branch;
}
public String getLoanId() { return loanId; }
public String getCustomerId() { return customerId; }
public BigDecimal getAmount() { return amount; }
public String getStatus() { return status; }
public String getBranch() { return branch; }
public void setLoanId(String loanId) { this.loanId = loanId; }
public void setCustomerId(String customerId) { this.customerId = customerId; }
public void setAmount(BigDecimal amount) { this.amount = amount; }
public void setStatus(String status) { this.status = status; }
public void setBranch(String branch) { this.branch = branch; }
@Override
public String toString() {
return loanId + "|" + customerId + "|" + amount + "|" + status + "|" + branch;
}
public static LoanApplication fromString(String value) {
String[] p = value.split("\\|");
return new LoanApplication(p[0], p[1], new BigDecimal(p[2]), p[3], p[4]);
}
}
LoanProcessorTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
import java.math.RoundingMode;
public class LoanProcessorTopology {
private static final BigDecimal INTEREST_RATE = new BigDecimal("0.085");
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> applications = builder.stream("loan-applications");
// Step 1: Log which thread processes each record
KStream<String, String> logged = applications.peek((key, value) -> {
String thread = Thread.currentThread().getName();
System.out.println("[" + thread + "] Processing loan: " + key);
});
// Step 2: Filter only approved loans
KStream<String, String> approved = logged
.filter((loanId, value) -> {
LoanApplication app = LoanApplication.fromString(value);
return "APPROVED".equals(app.getStatus());
});
// Step 3: Calculate interest
KStream<String, String> withInterest = approved
.mapValues(value -> {
LoanApplication app = LoanApplication.fromString(value);
BigDecimal interest = app.getAmount()
.multiply(INTEREST_RATE)
.setScale(2, RoundingMode.HALF_UP);
return app.toString() + "|INTEREST:" + interest;
});
// Step 4: Write to output topic
withInterest.to("processed-loans");
return builder.build();
}
}
LoanProcessorApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class LoanProcessorApp {
public static void main(String[] args) {
int numThreads = args.length > 0 ? Integer.parseInt(args[0]) : 2;
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, numThreads);
Topology topology = LoanProcessorTopology.build();
System.out.println("Topology:");
System.out.println(topology.describe());
KafkaStreams streams = new KafkaStreams(topology, props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Loan processor stopped");
}));
try {
streams.start();
System.out.println("Loan processor started");
System.out.println(" Stream threads: " + numThreads);
System.out.println(" Input topic: loan-applications");
System.out.println(" Output topic: processed-loans");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
How to run and observe thread distribution
# Step 1: Create topics with 6 partitions
kafka-topics.sh --create --topic loan-applications --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic processed-loans --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
# Step 2: Build
cd loan-processor
mvn clean package
# Step 3: Run with 2 threads
java -cp target/loan-processor-1.0.0.jar com.bank.streams.LoanProcessorApp 2
# Step 4: In another terminal, produce test loan applications
kafka-console-producer.sh --topic loan-applications --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type:
# LOAN001:LOAN001|CUST101|500000|APPROVED|Mumbai
# LOAN002:LOAN002|CUST102|300000|REJECTED|Delhi
# LOAN003:LOAN003|CUST103|750000|APPROVED|Chennai
# LOAN004:LOAN004|CUST104|200000|APPROVED|Kolkata
# Step 5: Observe which thread handles each record
# Step 6: Stop, restart with 6 threads, produce again, compare distribution
java -cp target/loan-processor-1.0.0.jar com.bank.streams.LoanProcessorApp 6
Expected output with 2 threads:
[loan-processor-StreamThread-1] Processing loan: LOAN001
[loan-processor-StreamThread-2] Processing loan: LOAN002
[loan-processor-StreamThread-1] Processing loan: LOAN003
[loan-processor-StreamThread-2] Processing loan: LOAN004
Expected output with 6 threads:
[loan-processor-StreamThread-1] Processing loan: LOAN001
[loan-processor-StreamThread-4] Processing loan: LOAN002
[loan-processor-StreamThread-2] Processing loan: LOAN003
[loan-processor-StreamThread-5] Processing loan: LOAN004
Expected output on processed-loans consumer:
LOAN001|CUST101|500000|APPROVED|Mumbai|INTEREST:42500.00
LOAN003|CUST103|750000|APPROVED|Chennai|INTEREST:63750.00
LOAN004|CUST104|200000|APPROVED|Kolkata|INTEREST:17000.00
LOAN002 (REJECTED) does not appear in the output.
What are common mistakes with Kafka Streams architecture?
Wrong: setting threads higher than partitions
package com.bank.streams;
public class WrongTooManyThreads {
public static void main(String[] args) {
System.out.println("WRONG: 6 partitions, 12 threads");
System.out.println();
System.out.println(" Threads 1-6: each gets 1 task, processing records");
System.out.println(" Threads 7-12: sit idle, no tasks to run");
System.out.println();
System.out.println(" Wasted resources:");
System.out.println(" - 6 idle OS threads consuming memory");
System.out.println(" - Each thread allocates buffers and network connections");
System.out.println(" - More GC pressure, no throughput gain");
}
}
Expected output:
WRONG: 6 partitions, 12 threads
Threads 1-6: each gets 1 task, processing records
Threads 7-12: sit idle, no tasks to run
Wasted resources:
- 6 idle OS threads consuming memory
- Each thread allocates buffers and network connections
- More GC pressure, no throughput gain
Correct: match threads to workload
package com.bank.streams;
public class CorrectThreadSizing {
public static void main(String[] args) {
System.out.println("CORRECT: Match threads to partition count");
System.out.println();
System.out.println(" Rule 1: num.stream.threads <= input partitions");
System.out.println(" Rule 2: Start with threads = CPU cores on the machine");
System.out.println(" Rule 3: If CPU-bound, threads = cores");
System.out.println(" Rule 4: If I/O-bound (state store reads), threads can exceed cores");
System.out.println(" Rule 5: Scale horizontally by adding instances, not just threads");
System.out.println();
System.out.println(" Example: 12 partitions, 4-core machines");
System.out.println(" Option A: 1 instance, 4 threads (each handles 3 tasks)");
System.out.println(" Option B: 3 instances, 4 threads each (each handles 1 task)");
System.out.println(" Option B gives better fault tolerance");
}
}
Expected output:
CORRECT: Match threads to partition count
Rule 1: num.stream.threads <= input partitions
Rule 2: Start with threads = CPU cores on the machine
Rule 3: If CPU-bound, threads = cores
Rule 4: If I/O-bound (state store reads), threads can exceed cores
Rule 5: Scale horizontally by adding instances, not just threads
Example: 12 partitions, 4-core machines
Option A: 1 instance, 4 threads (each handles 3 tasks)
Option B: 3 instances, 4 threads each (each handles 1 task)
Option B gives better fault tolerance
What should a senior or principal engineer know?
At the senior level, you need to size partitions, threads, and instances correctly for your workload. The partition count is set at topic creation and is hard to change later, so plan it based on expected throughput.
At the principal level, consider:
- Standby replicas: setting
num.standby.replicasto 1 or 2 keeps warm copies of state stores on other instances. On failover, the new owner does not need to rebuild state from the changelog topic, reducing recovery time from minutes to seconds. - State store locality: each task's state store lives on the instance that runs the task. If you reassign the task (rebalance), the state must be rebuilt from the changelog unless standby replicas are configured.
- Rack awareness: set
rack.idin the Streams config so Kafka assigns standby replicas to a different rack than the active task. - Cooperative rebalancing: since Kafka 2.4, Streams uses cooperative incremental rebalancing by default, which moves only the affected tasks instead of revoking all tasks during a rebalance.
- Topology optimization: enable
topology.optimization=allto let Kafka Streams merge repartition topics and reduce unnecessary intermediate topics. - Monitoring: expose JMX metrics for
stream-thread-metrics(commit-rate, poll-rate, process-rate) andstream-task-metrics(process-latency, commit-latency). Alert onREBALANCINGstate lasting more than 60 seconds.
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;
public class PrincipalConfig {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092,kafka-2:9092");
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4);
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2");
props.put(StreamsConfig.producerPrefix("acks"), "all");
props.put(StreamsConfig.RACK_AWARE_ASSIGNMENT_TAGS_CONFIG, "zone");
System.out.println("Principal-level Kafka Streams config:");
System.out.println(" Threads: 4");
System.out.println(" Standby replicas: 1 (warm failover)");
System.out.println(" Topology opt: OPTIMIZE (merge repartition topics)");
System.out.println(" Processing: exactly_once_v2");
System.out.println(" Producer acks: all");
System.out.println(" Rack awareness: enabled");
System.out.println();
System.out.println("Monitoring:");
System.out.println(" - stream-thread-metrics: commit-rate, poll-rate, process-rate");
System.out.println(" - stream-task-metrics: process-latency, commit-latency");
System.out.println(" - Alert: REBALANCING state > 60 seconds");
System.out.println(" - Alert: consumer lag > threshold per partition");
}
}
Expected output:
Principal-level Kafka Streams config:
Threads: 4
Standby replicas: 1 (warm failover)
Topology opt: OPTIMIZE (merge repartition topics)
Processing: exactly_once_v2
Producer acks: all
Rack awareness: enabled
Monitoring:
- stream-thread-metrics: commit-rate, poll-rate, process-rate
- stream-task-metrics: process-latency, commit-latency
- Alert: REBALANCING state > 60 seconds
- Alert: consumer lag > threshold per partition
What are common interview follow-up questions?
- What happens when a stream thread dies? (Its tasks are reassigned to surviving threads in the same instance or other instances. Stateful tasks rebuild state from changelog topics unless standby replicas are available.)
- Can two tasks from the same sub-topology run on different instances? (Yes. Tasks are distributed across all instances with the same application.id, just like consumer group partitions.)
- What is a standby task? (A standby task maintains a replica of a stateful task's state store on another instance. It continuously replays the changelog topic so it is ready for instant failover.)
- How do you increase parallelism for an existing topic? (Add partitions to the topic. Kafka Streams will detect the new partitions on the next rebalance and create additional tasks.)
- What is the difference between num.stream.threads and running more instances? (Threads add parallelism within one JVM; instances add parallelism across JVMs. Instances give better fault tolerance because a JVM crash loses only that instance's tasks.)
- What is cooperative rebalancing? (Instead of revoking all tasks during a rebalance, only the tasks that need to move are revoked. This reduces the stop-the-world pause and improves availability.)
Visual Overview
Kafka Streams Architecture
===========================
┌─────────────────────────────────────────┐
│ Kafka Cluster │
│ ┌─────────────────────────────────┐ │
│ │ loan-applications (6 partitions)│ │
│ │ P0 P1 P2 P3 P4 P5 │ │
│ └─────────────────────────────────┘ │
└────────────────┬────────────────────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│Instance A│ │Instance B│ │Instance C│
│ app.id: │ │ app.id: │ │ app.id: │
│ loan- │ │ loan- │ │ loan- │
│ processor│ │ processor│ │ processor│
│ │ │ │ │ │
│ Thread-1 │ │ Thread-1 │ │ Thread-1 │
│ Task 0_0│ │ Task 0_2│ │ Task 0_4│
│ Thread-2 │ │ Thread-2 │ │ Thread-2 │
│ Task 0_1│ │ Task 0_3│ │ Task 0_5│
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────┐
│ processed-loans (6 partitions) │
└─────────────────────────────────┘
Inside One Instance
====================
┌──────────────────────────────────────┐
│ JVM Instance │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Thread-1 │ │ Thread-2 │ │
│ │ │ │ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ Task 0_0 │ │ │ │ Task 0_2 │ │ │
│ │ │ P0 data │ │ │ │ P2 data │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ Task 0_1 │ │ │ │ Task 0_3 │ │ │
│ │ │ P1 data │ │ │ │ P3 data │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │
│ └──────────────┘ └──────────────┘ │
└──────────────────────────────────────┘
Key Takeaway
Kafka Streams architecture has three layers. At the top is the topology, which is the processing plan: source nodes read from topics, processor nodes transform data, and sink nodes write to topics. In the middle are stream tasks, one per input partition, each running its own copy of the topology. At the bottom are stream threads, which are Java threads that execute tasks.
The partition count is the hard limit on parallelism. If your topic has 6 partitions, Kafka Streams creates 6 tasks. You can run those tasks on 1 thread (sequentially) or 6 threads (one task per thread). Running more threads than partitions wastes resources because the extra threads have nothing to do.
In our National Bank of India loan processor, the loan-applications topic had 6 partitions. We ran 3 instances with 2 threads each, giving 6 threads total, one per task. Each thread logged which loan it processed, and the output showed even distribution. When we reduced to 1 instance with 2 threads, both threads handled 3 tasks each, and throughput dropped proportionally.
At the principal level, configure standby replicas so failover is instant, enable cooperative rebalancing to minimize stop-the-world pauses, use topology optimization to reduce intermediate topics, and monitor thread-level and task-level JMX metrics.
Remember one sentence: Kafka Streams creates one task per input partition, each task runs its own topology copy, and stream threads execute those tasks, so partition count equals maximum parallelism.
---