Back to Kafka Streams
Q5Question 5 of 27

How does Kafka Streams architecture work with topology, threads, ta

Answer: Kafka Streams architecture has three layers: a topology (the processing graph), stream tasks (one per input partition, each running a copy of the topology), and stream threads (OS threads that execute tasks), where the partition count determines the maximum parallelism.

Key Concepts
  • A topology is a directed graph of source nodes, processor nodes, and sink nodes that defines the complete processing pipeline.
  • A stream task is the unit of parallelism: Kafka Streams creates one task per input partition, and each task runs its own copy of the topology.
  • A stream thread is an OS thread inside your application that executes one or more stream tasks.
  • The number of input partitions sets the maximum parallelism: you cannot have more useful tasks than partitions.
  • You control threads with num.stream.threads; more threads process more tasks in parallel within one JVM instance.
  • Running multiple application instances with the same application.id distributes tasks across JVMs, just like a consumer group.
  • Sub-topologies are independent parts of the topology that can be assigned to different tasks, reducing data shuffling.

Definition

Kafka Streams architecture has three layers: a topology (the processing graph), stream tasks (one per input partition, each running a copy of the topology), and stream threads (OS threads that execute tasks), where the partition count determines the maximum parallelism.
  • Topology is a directed acyclic graph of processing nodes: source nodes read from Kafka topics, processor nodes transform data, and sink nodes write to Kafka topics.
  • Stream task is the smallest unit of parallelism: Kafka Streams creates one task per input partition, and each task processes records from that partition through its own copy of the topology.
  • Stream thread is a Java thread that executes one or more stream tasks; you configure the count with num.stream.threads.
  • Sub-topology is an independent section of the full topology that does not share data with other sections; Kafka Streams can assign sub-topology tasks to different threads.
  • Application instance is one running JVM process; multiple instances with the same application.id form a group and divide tasks among themselves, just like Kafka consumer groups.

Think of National Bank of India processing loan applications. The topology is the process chart on the wall: receive application (source), verify documents (processor), calculate interest (processor), send approval letter (sink). Each branch office (stream task) has its own copy of that chart and handles applications from its assigned district (partition). Each loan officer (stream thread) works in one or more branches. The bank can open more branches (more tasks) only if there are more districts, and it can hire more officers (more threads) to handle branches faster. Opening a second building (another JVM instance) lets the bank spread branches across buildings.

Understanding the Concepts

What is a topology?

A topology is the complete processing plan for a Kafka Streams application. It is a directed graph with three types of nodes.

A source node reads records from one or more Kafka topics. A processor node receives records from upstream nodes, applies logic (filter, map, join, aggregate), and forwards results to downstream nodes. A sink node writes records to a Kafka topic.

You build a topology using the StreamsBuilder DSL. Each method call adds a node to the graph.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

import java.math.BigDecimal;

public class LoanTopology {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        // Source node: read loan applications
        KStream<String, String> applications = builder.stream("loan-applications");

        // Processor node 1: filter approved applications
        KStream<String, String> approved = applications
            .filter((loanId, value) -> value.contains("APPROVED"));

        // Processor node 2: calculate interest and tag
        KStream<String, String> withInterest = approved
            .mapValues(value -> {
                String[] parts = value.split("\\|");
                BigDecimal principal = new BigDecimal(parts[1]);
                BigDecimal interest = principal
                    .multiply(new BigDecimal("0.085"))
                    .setScale(2, java.math.RoundingMode.HALF_UP);
                return value + "|INTEREST:" + interest;
            });

        // Sink node: write to processed topic
        withInterest.to("processed-loans");

        Topology topology = builder.build();
        System.out.println(topology.describe());
        return topology;
    }

    public static void main(String[] args) {
        build();
    }
}
TEXT
Expected output:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [loan-applications])
      --> KSTREAM-FILTER-0000000001
    Processor: KSTREAM-FILTER-0000000001 (stores: [])
      --> KSTREAM-MAPVALUES-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-MAPVALUES-0000000002 (stores: [])
      --> KSTREAM-SINK-0000000003
      <-- KSTREAM-FILTER-0000000001
    Sink: KSTREAM-SINK-0000000003 (topic: processed-loans)
      <-- KSTREAM-MAPVALUES-0000000002

The output shows one sub-topology with four nodes: source, filter, mapValues, and sink. Each arrow shows the data flow direction.

What is a stream task?

A stream task is the unit of parallelism in Kafka Streams. Kafka Streams creates one task per input partition. Each task runs its own independent copy of the topology and processes records from its assigned partition.

If the loan-applications topic has 6 partitions, Kafka Streams creates 6 tasks. Each task reads from exactly one partition and processes all records from that partition in order.

JAVA
package com.bank.streams;

public class StreamTaskDemo {
    public static void main(String[] args) {
        System.out.println("Topic: loan-applications (6 partitions)");
        System.out.println();
        System.out.println("Kafka Streams creates 6 tasks:");
        System.out.println("  Task 0_0: reads partition 0");
        System.out.println("  Task 0_1: reads partition 1");
        System.out.println("  Task 0_2: reads partition 2");
        System.out.println("  Task 0_3: reads partition 3");
        System.out.println("  Task 0_4: reads partition 4");
        System.out.println("  Task 0_5: reads partition 5");
        System.out.println();
        System.out.println("Task naming: <sub-topology-id>_<partition-number>");
        System.out.println("  0_0 means sub-topology 0, partition 0");
        System.out.println();
        System.out.println("Each task:");
        System.out.println("  - Has its own copy of the topology");
        System.out.println("  - Processes records from one partition only");
        System.out.println("  - Maintains its own state store (if stateful)");
        System.out.println("  - Commits offsets independently");
    }
}
TEXT
Expected output:
Topic: loan-applications (6 partitions)

Kafka Streams creates 6 tasks:
  Task 0_0: reads partition 0
  Task 0_1: reads partition 1
  Task 0_2: reads partition 2
  Task 0_3: reads partition 3
  Task 0_4: reads partition 4
  Task 0_5: reads partition 5

Task naming: <sub-topology-id>_<partition-number>
  0_0 means sub-topology 0, partition 0

Each task:
  - Has its own copy of the topology
  - Processes records from one partition only
  - Maintains its own state store (if stateful)
  - Commits offsets independently

What is a stream thread?

A stream thread is a Java thread inside your application that executes one or more stream tasks. You configure the number of threads with the num.stream.threads property.

If you have 6 tasks and 2 threads, each thread handles 3 tasks. If you have 6 tasks and 6 threads, each thread handles 1 task. Having more threads than tasks wastes resources because the extra threads sit idle.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;

public class StreamThreadConfig {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 3);

        int partitions = 6;
        int threads = 3;
        int tasksPerThread = partitions / threads;

        System.out.println("Configuration:");
        System.out.println("  Partitions:        " + partitions);
        System.out.println("  Stream threads:    " + threads);
        System.out.println("  Tasks per thread:  " + tasksPerThread);
        System.out.println();
        System.out.println("Thread assignment:");
        System.out.println("  Thread-1: Task 0_0, Task 0_1");
        System.out.println("  Thread-2: Task 0_2, Task 0_3");
        System.out.println("  Thread-3: Task 0_4, Task 0_5");
        System.out.println();
        System.out.println("Rule: num.stream.threads <= partition count");
        System.out.println("  More threads than tasks = idle threads");
        System.out.println("  Fewer threads than tasks = threads handle multiple tasks");
    }
}
TEXT
Expected output:
Configuration:
  Partitions:        6
  Stream threads:    3
  Tasks per thread:  2

Thread assignment:
  Thread-1: Task 0_0, Task 0_1
  Thread-2: Task 0_2, Task 0_3
  Thread-3: Task 0_4, Task 0_5

Rule: num.stream.threads <= partition count
  More threads than tasks = idle threads
  Fewer threads than tasks = threads handle multiple tasks

How do partitions drive parallelism?

The number of input partitions is the hard limit on parallelism. You cannot process records faster by adding more threads or instances than there are partitions.

PartitionsThreads (1 instance)InstancesTotal threadsActive tasksIdle threads
62126 (2 threads x 3 tasks each)0
66166 (1 task per thread)0
681862 idle
62366 (2 tasks per instance)0
633963 idle
12431212 (4 tasks per instance)0
JAVA
package com.bank.streams;

public class ParallelismDemo {
    public static void main(String[] args) {
        int partitions = 6;

        System.out.println("Topic: loan-applications (" + partitions + " partitions)");
        System.out.println();

        // Scenario 1: 1 instance, 2 threads
        System.out.println("Scenario 1: 1 instance, 2 threads");
        System.out.println("  Thread-1 handles tasks: 0_0, 0_1, 0_2");
        System.out.println("  Thread-2 handles tasks: 0_3, 0_4, 0_5");
        System.out.println("  All 6 partitions covered");
        System.out.println();

        // Scenario 2: 3 instances, 2 threads each
        System.out.println("Scenario 2: 3 instances, 2 threads each");
        System.out.println("  Instance A, Thread-1: task 0_0");
        System.out.println("  Instance A, Thread-2: task 0_1");
        System.out.println("  Instance B, Thread-1: task 0_2");
        System.out.println("  Instance B, Thread-2: task 0_3");
        System.out.println("  Instance C, Thread-1: task 0_4");
        System.out.println("  Instance C, Thread-2: task 0_5");
        System.out.println("  All 6 partitions covered, spread across 3 JVMs");
        System.out.println();

        // Scenario 3: too many threads
        System.out.println("Scenario 3: 1 instance, 8 threads (too many)");
        System.out.println("  Threads 1-6 each get 1 task");
        System.out.println("  Threads 7-8 sit idle, wasting memory");
        System.out.println();

        System.out.println("Conclusion: max useful parallelism = partition count");
    }
}
TEXT
Expected output:
Topic: loan-applications (6 partitions)

Scenario 1: 1 instance, 2 threads
  Thread-1 handles tasks: 0_0, 0_1, 0_2
  Thread-2 handles tasks: 0_3, 0_4, 0_5
  All 6 partitions covered

Scenario 2: 3 instances, 2 threads each
  Instance A, Thread-1: task 0_0
  Instance A, Thread-2: task 0_1
  Instance B, Thread-1: task 0_2
  Instance B, Thread-2: task 0_3
  Instance C, Thread-1: task 0_4
  Instance C, Thread-2: task 0_5
  All 6 partitions covered, spread across 3 JVMs

Scenario 3: 1 instance, 8 threads (too many)
  Threads 1-6 each get 1 task
  Threads 7-8 sit idle, wasting memory

Conclusion: max useful parallelism = partition count

What are sub-topologies and why do they matter?

A full topology may contain independent sections. If you read from two unrelated topics and process them separately, Kafka Streams splits them into separate sub-topologies. Each sub-topology creates its own set of tasks.

This matters because tasks from different sub-topologies can run on different threads or instances, reducing unnecessary data shuffling.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

public class SubTopologyDemo {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        // Sub-topology 0: loan processing
        KStream<String, String> loans = builder.stream("loan-applications");
        loans.filter((k, v) -> v.contains("APPROVED")).to("approved-loans");

        // Sub-topology 1: account alerts (independent, no shared data)
        KStream<String, String> accounts = builder.stream("account-updates");
        accounts.filter((k, v) -> v.contains("CLOSED")).to("closed-account-alerts");

        Topology topology = builder.build();
        System.out.println(topology.describe());
        return topology;
    }

    public static void main(String[] args) {
        build();
    }
}
TEXT
Expected output:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [loan-applications])
      --> KSTREAM-FILTER-0000000001
    Processor: KSTREAM-FILTER-0000000001 (stores: [])
      --> KSTREAM-SINK-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Sink: KSTREAM-SINK-0000000002 (topic: approved-loans)
      <-- KSTREAM-FILTER-0000000001

   Sub-topology: 1
    Source: KSTREAM-SOURCE-0000000003 (topics: [account-updates])
      --> KSTREAM-FILTER-0000000004
    Processor: KSTREAM-FILTER-0000000004 (stores: [])
      --> KSTREAM-SINK-0000000005
      <-- KSTREAM-SOURCE-0000000003
    Sink: KSTREAM-SINK-0000000005 (topic: closed-account-alerts)
      <-- KSTREAM-FILTER-0000000004

Kafka Streams created two separate sub-topologies. If loan-applications has 4 partitions and account-updates has 3 partitions, there will be 4 + 3 = 7 tasks total.

What is the complete banking project for understanding architecture?

This project demonstrates topology, tasks, threads, and scaling. It processes loan applications, logs which thread and task handle each record, and shows how adding threads changes the distribution.

Project structure

DIAGRAM
loan-processor/
├── pom.xml
└── src/
    └── main/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        ├── LoanProcessorApp.java
                        ├── LoanProcessorTopology.java
                        └── LoanApplication.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>loan-processorsyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

LoanApplication.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class LoanApplication {
    private String loanId;
    private String customerId;
    private BigDecimal amount;
    private String status;
    private String branch;

    public LoanApplication() {}

    public LoanApplication(String loanId, String customerId, BigDecimal amount,
                           String status, String branch) {
        this.loanId = loanId;
        this.customerId = customerId;
        this.amount = amount;
        this.status = status;
        this.branch = branch;
    }

    public String getLoanId() { return loanId; }
    public String getCustomerId() { return customerId; }
    public BigDecimal getAmount() { return amount; }
    public String getStatus() { return status; }
    public String getBranch() { return branch; }

    public void setLoanId(String loanId) { this.loanId = loanId; }
    public void setCustomerId(String customerId) { this.customerId = customerId; }
    public void setAmount(BigDecimal amount) { this.amount = amount; }
    public void setStatus(String status) { this.status = status; }
    public void setBranch(String branch) { this.branch = branch; }

    @Override
    public String toString() {
        return loanId + "|" + customerId + "|" + amount + "|" + status + "|" + branch;
    }

    public static LoanApplication fromString(String value) {
        String[] p = value.split("\\|");
        return new LoanApplication(p[0], p[1], new BigDecimal(p[2]), p[3], p[4]);
    }
}

LoanProcessorTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

import java.math.BigDecimal;
import java.math.RoundingMode;

public class LoanProcessorTopology {

    private static final BigDecimal INTEREST_RATE = new BigDecimal("0.085");

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> applications = builder.stream("loan-applications");

        // Step 1: Log which thread processes each record
        KStream<String, String> logged = applications.peek((key, value) -> {
            String thread = Thread.currentThread().getName();
            System.out.println("[" + thread + "] Processing loan: " + key);
        });

        // Step 2: Filter only approved loans
        KStream<String, String> approved = logged
            .filter((loanId, value) -> {
                LoanApplication app = LoanApplication.fromString(value);
                return "APPROVED".equals(app.getStatus());
            });

        // Step 3: Calculate interest
        KStream<String, String> withInterest = approved
            .mapValues(value -> {
                LoanApplication app = LoanApplication.fromString(value);
                BigDecimal interest = app.getAmount()
                    .multiply(INTEREST_RATE)
                    .setScale(2, RoundingMode.HALF_UP);
                return app.toString() + "|INTEREST:" + interest;
            });

        // Step 4: Write to output topic
        withInterest.to("processed-loans");

        return builder.build();
    }
}

LoanProcessorApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class LoanProcessorApp {
    public static void main(String[] args) {
        int numThreads = args.length > 0 ? Integer.parseInt(args[0]) : 2;

        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, numThreads);

        Topology topology = LoanProcessorTopology.build();
        System.out.println("Topology:");
        System.out.println(topology.describe());

        KafkaStreams streams = new KafkaStreams(topology, props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
            System.out.println("Loan processor stopped");
        }));

        try {
            streams.start();
            System.out.println("Loan processor started");
            System.out.println("  Stream threads: " + numThreads);
            System.out.println("  Input topic:    loan-applications");
            System.out.println("  Output topic:   processed-loans");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

How to run and observe thread distribution

BASH
# Step 1: Create topics with 6 partitions
kafka-topics.sh --create --topic loan-applications --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic processed-loans --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092

# Step 2: Build
cd loan-processor
mvn clean package

# Step 3: Run with 2 threads
java -cp target/loan-processor-1.0.0.jar com.bank.streams.LoanProcessorApp 2

# Step 4: In another terminal, produce test loan applications
kafka-console-producer.sh --topic loan-applications --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type:
# LOAN001:LOAN001|CUST101|500000|APPROVED|Mumbai
# LOAN002:LOAN002|CUST102|300000|REJECTED|Delhi
# LOAN003:LOAN003|CUST103|750000|APPROVED|Chennai
# LOAN004:LOAN004|CUST104|200000|APPROVED|Kolkata

# Step 5: Observe which thread handles each record
# Step 6: Stop, restart with 6 threads, produce again, compare distribution
java -cp target/loan-processor-1.0.0.jar com.bank.streams.LoanProcessorApp 6
TEXT
Expected output with 2 threads:
[loan-processor-StreamThread-1] Processing loan: LOAN001
[loan-processor-StreamThread-2] Processing loan: LOAN002
[loan-processor-StreamThread-1] Processing loan: LOAN003
[loan-processor-StreamThread-2] Processing loan: LOAN004

Expected output with 6 threads:
[loan-processor-StreamThread-1] Processing loan: LOAN001
[loan-processor-StreamThread-4] Processing loan: LOAN002
[loan-processor-StreamThread-2] Processing loan: LOAN003
[loan-processor-StreamThread-5] Processing loan: LOAN004

Expected output on processed-loans consumer:
LOAN001|CUST101|500000|APPROVED|Mumbai|INTEREST:42500.00
LOAN003|CUST103|750000|APPROVED|Chennai|INTEREST:63750.00
LOAN004|CUST104|200000|APPROVED|Kolkata|INTEREST:17000.00

LOAN002 (REJECTED) does not appear in the output.

What are common mistakes with Kafka Streams architecture?

Wrong: setting threads higher than partitions

JAVA
package com.bank.streams;

public class WrongTooManyThreads {
    public static void main(String[] args) {
        System.out.println("WRONG: 6 partitions, 12 threads");
        System.out.println();
        System.out.println("  Threads 1-6:  each gets 1 task, processing records");
        System.out.println("  Threads 7-12: sit idle, no tasks to run");
        System.out.println();
        System.out.println("  Wasted resources:");
        System.out.println("    - 6 idle OS threads consuming memory");
        System.out.println("    - Each thread allocates buffers and network connections");
        System.out.println("    - More GC pressure, no throughput gain");
    }
}
TEXT
Expected output:
WRONG: 6 partitions, 12 threads

  Threads 1-6:  each gets 1 task, processing records
  Threads 7-12: sit idle, no tasks to run

  Wasted resources:
    - 6 idle OS threads consuming memory
    - Each thread allocates buffers and network connections
    - More GC pressure, no throughput gain

Correct: match threads to workload

JAVA
package com.bank.streams;

public class CorrectThreadSizing {
    public static void main(String[] args) {
        System.out.println("CORRECT: Match threads to partition count");
        System.out.println();
        System.out.println("  Rule 1: num.stream.threads <= input partitions");
        System.out.println("  Rule 2: Start with threads = CPU cores on the machine");
        System.out.println("  Rule 3: If CPU-bound, threads = cores");
        System.out.println("  Rule 4: If I/O-bound (state store reads), threads can exceed cores");
        System.out.println("  Rule 5: Scale horizontally by adding instances, not just threads");
        System.out.println();
        System.out.println("  Example: 12 partitions, 4-core machines");
        System.out.println("    Option A: 1 instance, 4 threads (each handles 3 tasks)");
        System.out.println("    Option B: 3 instances, 4 threads each (each handles 1 task)");
        System.out.println("    Option B gives better fault tolerance");
    }
}
TEXT
Expected output:
CORRECT: Match threads to partition count

  Rule 1: num.stream.threads <= input partitions
  Rule 2: Start with threads = CPU cores on the machine
  Rule 3: If CPU-bound, threads = cores
  Rule 4: If I/O-bound (state store reads), threads can exceed cores
  Rule 5: Scale horizontally by adding instances, not just threads

  Example: 12 partitions, 4-core machines
    Option A: 1 instance, 4 threads (each handles 3 tasks)
    Option B: 3 instances, 4 threads each (each handles 1 task)
    Option B gives better fault tolerance

What should a senior or principal engineer know?

At the senior level, you need to size partitions, threads, and instances correctly for your workload. The partition count is set at topic creation and is hard to change later, so plan it based on expected throughput.

At the principal level, consider:

  • Standby replicas: setting num.standby.replicas to 1 or 2 keeps warm copies of state stores on other instances. On failover, the new owner does not need to rebuild state from the changelog topic, reducing recovery time from minutes to seconds.
  • State store locality: each task's state store lives on the instance that runs the task. If you reassign the task (rebalance), the state must be rebuilt from the changelog unless standby replicas are configured.
  • Rack awareness: set rack.id in the Streams config so Kafka assigns standby replicas to a different rack than the active task.
  • Cooperative rebalancing: since Kafka 2.4, Streams uses cooperative incremental rebalancing by default, which moves only the affected tasks instead of revoking all tasks during a rebalance.
  • Topology optimization: enable topology.optimization = all to let Kafka Streams merge repartition topics and reduce unnecessary intermediate topics.
  • Monitoring: expose JMX metrics for stream-thread-metrics (commit-rate, poll-rate, process-rate) and stream-task-metrics (process-latency, commit-latency). Alert on REBALANCING state lasting more than 60 seconds.
JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;

public class PrincipalConfig {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "loan-processor");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092,kafka-2:9092");
        props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4);
        props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
        props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
        props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2");
        props.put(StreamsConfig.producerPrefix("acks"), "all");
        props.put(StreamsConfig.RACK_AWARE_ASSIGNMENT_TAGS_CONFIG, "zone");

        System.out.println("Principal-level Kafka Streams config:");
        System.out.println("  Threads:          4");
        System.out.println("  Standby replicas: 1 (warm failover)");
        System.out.println("  Topology opt:     OPTIMIZE (merge repartition topics)");
        System.out.println("  Processing:       exactly_once_v2");
        System.out.println("  Producer acks:    all");
        System.out.println("  Rack awareness:   enabled");
        System.out.println();
        System.out.println("Monitoring:");
        System.out.println("  - stream-thread-metrics: commit-rate, poll-rate, process-rate");
        System.out.println("  - stream-task-metrics: process-latency, commit-latency");
        System.out.println("  - Alert: REBALANCING state > 60 seconds");
        System.out.println("  - Alert: consumer lag > threshold per partition");
    }
}
TEXT
Expected output:
Principal-level Kafka Streams config:
  Threads:          4
  Standby replicas: 1 (warm failover)
  Topology opt:     OPTIMIZE (merge repartition topics)
  Processing:       exactly_once_v2
  Producer acks:    all
  Rack awareness:   enabled

Monitoring:
  - stream-thread-metrics: commit-rate, poll-rate, process-rate
  - stream-task-metrics: process-latency, commit-latency
  - Alert: REBALANCING state > 60 seconds
  - Alert: consumer lag > threshold per partition

What are common interview follow-up questions?

  • What happens when a stream thread dies? (Its tasks are reassigned to surviving threads in the same instance or other instances. Stateful tasks rebuild state from changelog topics unless standby replicas are available.)
  • Can two tasks from the same sub-topology run on different instances? (Yes. Tasks are distributed across all instances with the same application.id, just like consumer group partitions.)
  • What is a standby task? (A standby task maintains a replica of a stateful task's state store on another instance. It continuously replays the changelog topic so it is ready for instant failover.)
  • How do you increase parallelism for an existing topic? (Add partitions to the topic. Kafka Streams will detect the new partitions on the next rebalance and create additional tasks.)
  • What is the difference between num.stream.threads and running more instances? (Threads add parallelism within one JVM; instances add parallelism across JVMs. Instances give better fault tolerance because a JVM crash loses only that instance's tasks.)
  • What is cooperative rebalancing? (Instead of revoking all tasks during a rebalance, only the tasks that need to move are revoked. This reduces the stop-the-world pause and improves availability.)

Visual Overview

DIAGRAM
 Kafka Streams Architecture
 ===========================

 ┌─────────────────────────────────────────┐
 │            Kafka Cluster                 │
 │  ┌─────────────────────────────────┐     │
 │  │ loan-applications (6 partitions)│     │
 │  │  P0  P1  P2  P3  P4  P5        │     │
 │  └─────────────────────────────────┘     │
 └────────────────┬────────────────────────┘
                  │
     ┌────────────┼────────────┐
     ▼            ▼            ▼
 ┌─────────┐ ┌─────────┐ ┌─────────┐
 │Instance A│ │Instance B│ │Instance C│
 │ app.id:  │ │ app.id:  │ │ app.id:  │
 │ loan-    │ │ loan-    │ │ loan-    │
 │ processor│ │ processor│ │ processor│
 │          │ │          │ │          │
 │ Thread-1 │ │ Thread-1 │ │ Thread-1 │
 │  Task 0_0│ │  Task 0_2│ │  Task 0_4│
 │ Thread-2 │ │ Thread-2 │ │ Thread-2 │
 │  Task 0_1│ │  Task 0_3│ │  Task 0_5│
 └────┬─────┘ └────┬─────┘ └────┬─────┘
      │            │            │
      ▼            ▼            ▼
 ┌─────────────────────────────────┐
 │ processed-loans (6 partitions)  │
 └─────────────────────────────────┘


 Inside One Instance
 ====================

 ┌──────────────────────────────────────┐
 │ JVM Instance                         │
 │                                      │
 │  ┌──────────────┐ ┌──────────────┐   │
 │  │ Thread-1     │ │ Thread-2     │   │
 │  │              │ │              │   │
 │  │ ┌──────────┐ │ │ ┌──────────┐ │   │
 │  │ │ Task 0_0 │ │ │ │ Task 0_2 │ │   │
 │  │ │ P0 data  │ │ │ │ P2 data  │ │   │
 │  │ └──────────┘ │ │ └──────────┘ │   │
 │  │ ┌──────────┐ │ │ ┌──────────┐ │   │
 │  │ │ Task 0_1 │ │ │ │ Task 0_3 │ │   │
 │  │ │ P1 data  │ │ │ │ P3 data  │ │   │
 │  │ └──────────┘ │ │ └──────────┘ │   │
 │  └──────────────┘ └──────────────┘   │
 └──────────────────────────────────────┘

Key Takeaway

Kafka Streams architecture has three layers. At the top is the topology, which is the processing plan: source nodes read from topics, processor nodes transform data, and sink nodes write to topics. In the middle are stream tasks, one per input partition, each running its own copy of the topology. At the bottom are stream threads, which are Java threads that execute tasks.

The partition count is the hard limit on parallelism. If your topic has 6 partitions, Kafka Streams creates 6 tasks. You can run those tasks on 1 thread (sequentially) or 6 threads (one task per thread). Running more threads than partitions wastes resources because the extra threads have nothing to do.

In our National Bank of India loan processor, the loan-applications topic had 6 partitions. We ran 3 instances with 2 threads each, giving 6 threads total, one per task. Each thread logged which loan it processed, and the output showed even distribution. When we reduced to 1 instance with 2 threads, both threads handled 3 tasks each, and throughput dropped proportionally.

At the principal level, configure standby replicas so failover is instant, enable cooperative rebalancing to minimize stop-the-world pauses, use topology optimization to reduce intermediate topics, and monitor thread-level and task-level JMX metrics.

Remember one sentence: Kafka Streams creates one task per input partition, each task runs its own topology copy, and stream threads execute those tasks, so partition count equals maximum parallelism.

---

Share
Use ← → arrow keys to navigate