Definition
- Kafka Streams is a client library bundled with Apache Kafka for building stream-processing applications; it runs inside your JVM process, not on a separate cluster like Flink or Spark.
- Consumer API is the low-level Kafka client that polls records from topics; you write your own processing loop, manage offsets, handle errors, and coordinate state yourself.
- Topology is the processing plan in Kafka Streams: a directed graph of source nodes (read), processor nodes (transform), and sink nodes (write).
- State store is the local storage Kafka Streams uses for stateful operations like aggregations; it is backed by a changelog topic for fault tolerance.
- Exactly-once processing is built into Kafka Streams via Kafka transactions; with the Consumer API you must implement idempotency yourself.
Think of National Bank of India processing daily transactions. The Consumer API is like a clerk who picks up one paper slip at a time from a pile, reads it, and manually decides what to do: calculate the running balance, check for fraud, file the result. The clerk tracks which slips have been read and handles errors alone. Kafka Streams is like an automated conveyor belt system in the same bank: slips enter at one end, pass through a series of stations (filter suspicious amounts, calculate running balance, route to the right department), and exit at the other end. The conveyor handles slip tracking, station recovery, and parallel lanes automatically.
Understanding the Concepts
What is Kafka Streams?
Kafka Streams is a Java library provided by Apache Kafka for processing data from Kafka topics in real time. You add it as a Maven or Gradle dependency, write your processing logic, and run it as a normal Java application. There is no separate cluster to deploy or manage.
The core idea is a topology: you define a pipeline of operations (read from a topic, filter, transform, aggregate, write to another topic), and Kafka Streams executes that pipeline continuously.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class WhatIsKafkaStreams {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Read transactions from a Kafka topic
KStream<String, String> transactions = builder.stream("bank-transactions");
// Filter only high-value transactions
KStream<String, String> highValue = transactions
.filter((accountId, txn) -> {
// In a real app, parse the value and check amount
System.out.println("Filtering transaction for account: " + accountId);
return txn.contains("HIGH");
});
// Write filtered transactions to another topic
highValue.to("high-value-alerts");
Topology topology = builder.build();
System.out.println("Topology description:");
System.out.println(topology.describe());
}
}
Expected output:
Topology description:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-SINK-0000000002
<-- KSTREAM-SOURCE-0000000000
Sink: KSTREAM-SINK-0000000002 (topic: high-value-alerts)
<-- KSTREAM-FILTER-0000000001
What is the Kafka Consumer API?
The Consumer API is the low-level client that reads records from Kafka topics. You create a KafkaConsumer, subscribe to topics, call poll() in a loop, and process each record yourself. You manage offsets, error handling, and any state.
package com.bank.consumer;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class TransactionConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "bank-transaction-processor");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("bank-transactions"));
System.out.println("Consumer started, polling bank-transactions...");
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord<String, String> record : records) {
// All processing logic is manual
String accountId = record.key();
String txn = record.value();
if (txn.contains("HIGH")) {
System.out.println("HIGH value alert: account=" + accountId + ", txn=" + txn);
// Manually write to another topic or call a service
}
}
consumer.commitSync();
}
} finally {
consumer.close();
System.out.println("Consumer closed");
}
}
}
Expected output:
Consumer started, polling bank-transactions...
HIGH value alert: account=SAV2001, txn=DEBIT 500000 HIGH
Consumer closed
How does Kafka Streams differ from the Consumer API?
The difference is in what the library handles for you versus what you handle yourself.
| Feature | Kafka Consumer API | Kafka Streams |
|---|---|---|
| Processing model | Manual poll loop | Declarative topology |
| Offset management | You call commitSync/commitAsync | Automatic |
| State management | You build and maintain state | Built-in state stores (RocksDB) |
| Fault tolerance | You implement retry and recovery | Automatic via changelog topics |
| Exactly-once | You implement idempotency | Built-in (processing.guarantee) |
| Transformations | You write filter/map/join logic | filter(), map(), join(), aggregate() |
| Windowing | You implement time bucketing | Built-in tumbling, hopping, session windows |
| Writing output | You create a separate KafkaProducer | .to() writes directly in the topology |
| Scaling | You manage consumer group rebalancing | Automatic partition assignment |
| Testing | You mock the consumer | TopologyTestDriver (no broker needed) |
package com.bank.streams;
public class ConsumerVsStreamsComparison {
public static void main(String[] args) {
System.out.println("=== Consumer API ===");
System.out.println("1. Create KafkaConsumer with properties");
System.out.println("2. Subscribe to topic");
System.out.println("3. Poll in a loop");
System.out.println("4. Process each record manually");
System.out.println("5. Commit offsets manually");
System.out.println("6. Create KafkaProducer to write output");
System.out.println("7. Handle errors, retries, state yourself");
System.out.println();
System.out.println("=== Kafka Streams ===");
System.out.println("1. Create StreamsBuilder");
System.out.println("2. Define topology: stream -> filter -> map -> to");
System.out.println("3. Start KafkaStreams instance");
System.out.println("4. Offsets, state, fault tolerance are automatic");
System.out.println("5. Output writing is part of the topology");
System.out.println();
System.out.println("Consumer API: 7 manual steps");
System.out.println("Kafka Streams: 3 steps, rest is automatic");
}
}
Expected output:
=== Consumer API ===
1. Create KafkaConsumer with properties
2. Subscribe to topic
3. Poll in a loop
4. Process each record manually
5. Commit offsets manually
6. Create KafkaProducer to write output
7. Handle errors, retries, state yourself
=== Kafka Streams ===
1. Create StreamsBuilder
2. Define topology: stream -> filter -> map -> to
3. Start KafkaStreams instance
4. Offsets, state, fault tolerance are automatic
5. Output writing is part of the topology
Consumer API: 7 manual steps
Kafka Streams: 3 steps, rest is automatic
When should you use the Consumer API instead of Kafka Streams?
The Consumer API is the right choice when you only need to read messages and call an external service, without writing results back to Kafka. For example, reading a payment notification and sending an email does not need a topology.
Another case is when your processing requires calling a database or REST API for every record. Kafka Streams is designed for Kafka-in, Kafka-out pipelines. If you need to call an external system in the middle, the Consumer API gives you more control over error handling and retries.
package com.bank.consumer;
public class WhenToUseConsumerApi {
public static void main(String[] args) {
System.out.println("Use Consumer API when:");
System.out.println(" 1. You read from Kafka and call an external service (email, SMS, REST API)");
System.out.println(" 2. You need fine-grained control over offset commits");
System.out.println(" 3. Your processing is simple: read, act, done");
System.out.println(" 4. You do not write results back to Kafka");
System.out.println();
System.out.println("Use Kafka Streams when:");
System.out.println(" 1. You read from Kafka, process, and write back to Kafka");
System.out.println(" 2. You need filter, map, join, aggregate, or windowing");
System.out.println(" 3. You need stateful processing with automatic fault tolerance");
System.out.println(" 4. You need exactly-once guarantees in the pipeline");
}
}
Expected output:
Use Consumer API when:
1. You read from Kafka and call an external service (email, SMS, REST API)
2. You need fine-grained control over offset commits
3. Your processing is simple: read, act, done
4. You do not write results back to Kafka
Use Kafka Streams when:
1. You read from Kafka, process, and write back to Kafka
2. You need filter, map, join, aggregate, or windowing
3. You need stateful processing with automatic fault tolerance
4. You need exactly-once guarantees in the pipeline
What is a complete banking project using Kafka Streams?
This project reads payment transactions from a Kafka topic, filters transactions above 100,000 rupees, normalizes the description to uppercase, and writes the results to an alerts topic. A junior developer can run this project from start to finish.
Project structure
payment-alert-service/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── PaymentAlertApp.java (main class)
├── PaymentAlertTopology.java (topology definition)
└── PaymentTransaction.java (model)
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>payment-alert-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
PaymentTransaction.java
package com.bank.streams;
import java.math.BigDecimal;
public class PaymentTransaction {
private String accountId;
private String description;
private BigDecimal amount;
private String status;
public PaymentTransaction() {}
public PaymentTransaction(String accountId, String description,
BigDecimal amount, String status) {
this.accountId = accountId;
this.description = description;
this.amount = amount;
this.status = status;
}
public String getAccountId() { return accountId; }
public void setAccountId(String accountId) { this.accountId = accountId; }
public String getDescription() { return description; }
public void setDescription(String description) { this.description = description; }
public BigDecimal getAmount() { return amount; }
public void setAmount(BigDecimal amount) { this.amount = amount; }
public String getStatus() { return status; }
public void setStatus(String status) { this.status = status; }
@Override
public String toString() {
return accountId + "|" + description + "|" + amount + "|" + status;
}
public static PaymentTransaction fromString(String value) {
String[] parts = value.split("\\|");
return new PaymentTransaction(
parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
PaymentAlertTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
public class PaymentAlertTopology {
private static final BigDecimal ALERT_THRESHOLD = new BigDecimal("100000");
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// Source: read raw payment events
KStream<String, String> payments = builder.stream("payments");
// Step 1: Parse the payment string into fields
// Step 2: Filter transactions above 1,00,000 rupees
// Step 3: Normalize description to uppercase
// Step 4: Write to alerts topic
payments
.filter((key, value) -> {
PaymentTransaction txn = PaymentTransaction.fromString(value);
return txn.getAmount().compareTo(ALERT_THRESHOLD) > 0;
})
.mapValues(value -> {
PaymentTransaction txn = PaymentTransaction.fromString(value);
txn.setDescription(txn.getDescription().toUpperCase());
return txn.toString();
})
.to("high-value-alerts");
return builder.build();
}
}
PaymentAlertApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class PaymentAlertApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-alert-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
PaymentAlertTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Payment alert service shut down");
}));
try {
streams.start();
System.out.println("Payment alert service started");
System.out.println(" Input topic: payments");
System.out.println(" Output topic: high-value-alerts");
System.out.println(" Threshold: 100000 INR");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
Expected output (on startup):
Payment alert service started
Input topic: payments
Output topic: high-value-alerts
Threshold: 100000 INR
How to run the project
# Step 1: Start Kafka (using Docker Compose or local install)
docker-compose up -d kafka zookeeper
# Step 2: Create the input and output topics
kafka-topics.sh --create --topic payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic high-value-alerts --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build the project
cd payment-alert-service
mvn clean package
# Step 4: Run the Kafka Streams app
java -cp target/payment-alert-service-1.0.0.jar com.bank.streams.PaymentAlertApp
# Step 5: In a new terminal, produce test messages
kafka-console-producer.sh --topic payments --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type these messages (key:value format):
# SAV2001:SAV2001|neft transfer to Priya|250000|SUCCESS
# SAV2002:SAV2002|atm withdrawal|5000|SUCCESS
# CUR3001:CUR3001|rtgs payment to vendor|500000|SUCCESS
# Step 6: In another terminal, consume the output topic
kafka-console-consumer.sh --topic high-value-alerts --from-beginning --bootstrap-server localhost:9092
Expected output on the high-value-alerts consumer:
SAV2001|NEFT TRANSFER TO PRIYA|250000|SUCCESS
CUR3001|RTGS PAYMENT TO VENDOR|500000|SUCCESS
Notice that SAV2002 (5,000 rupees) did not appear because it is below the 100,000 threshold. Only the two high-value payments passed through, and their descriptions are now uppercase.
What are common mistakes when choosing between Consumer API and Kafka Streams?
The most common mistake is using the Consumer API for complex processing that Kafka Streams handles better.
Wrong: manual aggregation with Consumer API
package com.bank.consumer;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.Map;
public class WrongManualAggregation {
public static void main(String[] args) {
// Simulating what developers do wrong: manual state in a HashMap
Map<String, BigDecimal> balances = new HashMap<>();
// Problem 1: This state is lost if the app restarts
balances.merge("SAV2001", new BigDecimal("50000"), BigDecimal::add);
balances.merge("SAV2001", new BigDecimal("30000"), BigDecimal::add);
balances.merge("SAV2002", new BigDecimal("10000"), BigDecimal::add);
System.out.println("Manual aggregation (WRONG approach):");
balances.forEach((account, balance) ->
System.out.println(" " + account + ": " + balance));
System.out.println();
System.out.println("Problems with this approach:");
System.out.println(" 1. State is in-memory only, lost on restart");
System.out.println(" 2. No fault tolerance, no changelog backup");
System.out.println(" 3. No exactly-once guarantee");
System.out.println(" 4. Scaling requires manual partition handling");
}
}
Expected output:
Manual aggregation (WRONG approach):
SAV2001: 80000
SAV2002: 10000
Problems with this approach:
1. State is in-memory only, lost on restart
2. No fault tolerance, no changelog backup
3. No exactly-once guarantee
4. Scaling requires manual partition handling
Correct: use Kafka Streams aggregate with state store
package com.bank.streams;
public class CorrectStreamsAggregation {
public static void main(String[] args) {
System.out.println("Kafka Streams aggregation (CORRECT approach):");
System.out.println(" KStream -> groupByKey() -> aggregate()");
System.out.println();
System.out.println("What Kafka Streams handles automatically:");
System.out.println(" 1. State stored in RocksDB, survives restarts");
System.out.println(" 2. Changelog topic backs up state for recovery");
System.out.println(" 3. exactly_once_v2 makes read-process-write atomic");
System.out.println(" 4. Scaling: just start more instances, partitions rebalance");
}
}
Expected output:
Kafka Streams aggregation (CORRECT approach):
KStream -> groupByKey() -> aggregate()
What Kafka Streams handles automatically:
1. State stored in RocksDB, survives restarts
2. Changelog topic backs up state for recovery
3. exactly_once_v2 makes read-process-write atomic
4. Scaling: just start more instances, partitions rebalance
What about testing the topology without a Kafka broker?
Kafka Streams provides TopologyTestDriver for unit testing. You create the topology, feed test records into a TestInputTopic, and read results from a TestOutputTopic. No running broker is needed.
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class PaymentAlertTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(PaymentAlertTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"payments", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"high-value-alerts", new StringDeserializer(), new StringDeserializer());
// Test 1: High value transaction (should pass through)
input.pipeInput("SAV2001", "SAV2001|neft transfer to Priya|250000|SUCCESS");
// Test 2: Low value transaction (should be filtered out)
input.pipeInput("SAV2002", "SAV2002|atm withdrawal|5000|SUCCESS");
// Test 3: Another high value transaction
input.pipeInput("CUR3001", "CUR3001|rtgs payment|500000|SUCCESS");
System.out.println("Records in output topic: " + output.getQueueSize());
System.out.println("Record 1: " + output.readValue());
System.out.println("Record 2: " + output.readValue());
System.out.println("Queue empty: " + output.isEmpty());
System.out.println();
System.out.println("Test passed: 2 high-value, 0 low-value in output");
}
}
}
Expected output:
Records in output topic: 2
Record 1: SAV2001|NEFT TRANSFER TO PRIYA|250000|SUCCESS
Record 2: CUR3001|RTGS PAYMENT|500000|SUCCESS
Queue empty: true
Test passed: 2 high-value, 0 low-value in output
What should a senior or principal engineer know?
At the senior level, you should understand when Kafka Streams is the right tool and when it is not. Kafka Streams is designed for Kafka-to-Kafka pipelines. If your output is a database, REST call, or email, the Consumer API with a service layer is often simpler.
At the principal level, you need to think about:
- Topology isolation: each Kafka Streams application should own its topology. Do not put unrelated processing into one app.
- Partition count determines parallelism: the maximum number of stream threads doing useful work equals the number of input partitions. Plan partition count at topic creation time.
- State store sizing: stateful operations create local RocksDB stores. On restart, state is rebuilt from changelog topics, which takes time proportional to state size.
- Exactly-once overhead:
exactly_once_v2uses Kafka transactions, which reduce throughput by 10-30%. Measure before enabling. - Upgrade strategy: Kafka Streams apps are just JVM processes, so rolling restarts work. But topology changes (adding a new processor) require careful planning because stream thread rebalancing happens.
- Monitoring: expose JMX metrics for consumer lag, state store size, and stream thread state. Alert on
REBALANCINGstate lasting more than 60 seconds.
package com.bank.streams;
public class SeniorKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal Kafka Streams checklist:");
System.out.println(" 1. Partition count = max parallelism");
System.out.println(" 2. State stores use RocksDB, rebuilt from changelog on restart");
System.out.println(" 3. exactly_once_v2 reduces throughput 10-30%");
System.out.println(" 4. One topology per app, do not mix unrelated logic");
System.out.println(" 5. Monitor consumer lag, stream-thread-state, state-store-size");
System.out.println(" 6. Rolling restart for upgrades, plan for rebalancing time");
System.out.println(" 7. Use TopologyTestDriver for unit tests, EmbeddedKafka for integration");
}
}
Expected output:
Senior/Principal Kafka Streams checklist:
1. Partition count = max parallelism
2. State stores use RocksDB, rebuilt from changelog on restart
3. exactly_once_v2 reduces throughput 10-30%
4. One topology per app, do not mix unrelated logic
5. Monitor consumer lag, stream-thread-state, state-store-size
6. Rolling restart for upgrades, plan for rebalancing time
7. Use TopologyTestDriver for unit tests, EmbeddedKafka for integration
What are common interview follow-up questions?
- Can Kafka Streams call a REST API or database during processing? (Yes, but it blocks the stream thread and is discouraged. Use the Consumer API for external calls.)
- What happens if a Kafka Streams app crashes mid-processing? (With exactly_once_v2, uncommitted work is rolled back. On restart, processing resumes from the last committed offset.)
- Can you run Kafka Streams in a Spring Boot application? (Yes. Spring Kafka provides auto-configuration for Kafka Streams with
@EnableKafkaStreams.) - How many stream threads should you configure? (One thread processes one or more tasks, where each task corresponds to a partition. Default is one thread. Set
num.stream.threadsto match CPU cores, but never more than input partitions.) - Is Kafka Streams the same as KSQL? (No. KSQL/ksqlDB is a SQL layer on top of Kafka Streams. Kafka Streams is the Java library underneath.)
Visual Overview
Kafka Consumer API Flow
========================
┌──────────────┐
│ Kafka Topic │
│ (payments) │
└──────┬───────┘
│ poll()
▼
┌──────────────────────┐
│ KafkaConsumer │
│ for each record: │
│ parse │
│ filter manually │
│ process manually │
│ produce manually │
│ commitSync() │
└──────────────────────┘
Kafka Streams Flow
===================
┌──────────────┐
│ Source Topic │
│ (payments) │
└──────┬───────┘
│
▼
┌──────────────────────┐
│ Filter Node │
│ amount > 100,000 │
└──────┬───────────────┘
│
▼
┌──────────────────────┐
│ MapValues Node │
│ uppercase description│
└──────┬───────────────┘
│
▼
┌──────────────────────┐
│ Sink Topic │
│ (high-value-alerts) │
└──────────────────────┘
Offsets, state, and recovery are automatic.
Key Takeaway
Kafka Streams is a Java library that runs inside your application and processes data from Kafka topics in real time. Unlike the Consumer API, where you write a polling loop and manage offsets, state, and error handling yourself, Kafka Streams lets you define a topology of operations and handles the rest.
The Consumer API is the right choice when you need to read a message and call an external service like a REST API, send an email, or write to a database. Kafka Streams is the right choice when your pipeline is Kafka-in, Kafka-out: read from a topic, filter, transform, aggregate, and write to another topic.
In our National Bank of India payment-alert-service project, we read payment transactions, filtered amounts above one lakh rupees, normalized descriptions to uppercase, and wrote alerts to a separate topic. The entire pipeline was three Kafka Streams operations: filter, mapValues, and to. We tested it with TopologyTestDriver without starting a broker.
At the senior level, remember that partition count determines parallelism, state stores are backed by changelog topics, and exactly-once processing uses Kafka transactions with a measurable throughput cost. At the principal level, design one topology per application, monitor consumer lag and stream-thread state, and plan for rebalancing time during rolling upgrades.
Remember one sentence: Kafka Streams is a library, not a server, that turns a topology of filter, map, join, and aggregate into a fault-tolerant, scalable, exactly-once pipeline running inside your own JVM.
---