Definition
- KStream represents an event stream: each record is a fact that happened (a transaction occurred, a payment was made, a login was recorded), and duplicate keys mean multiple events, not updates.
- Unbounded means the stream has no end: records keep arriving as long as producers publish to the topic.
- Record independence means each record is processed on its own; a new record with key SAV2001 does not overwrite the previous SAV2001 record.
- Stateless operations on KStream (filter, map, mapValues, flatMap, peek, branch, merge) process each record without needing memory of prior records.
- Stateful transition: calling groupByKey() on a KStream produces a KGroupedStream, and then calling aggregate(), reduce(), or count() produces a KTable with accumulated state.
Think of the transaction log at National Bank of India. Every time Rajesh deposits money, withdraws money, or transfers money, a new line is written in the log. The log never erases old entries. If Rajesh deposits 50,000 rupees at 10:00 AM and another 30,000 rupees at 11:00 AM, both lines exist. That is a KStream: a continuous log of events. The bank does not overwrite the 10:00 AM entry with the 11:00 AM entry. Each deposit is a separate fact. If Priya at the counter wants to know Rajesh's current balance, she must add up all the events. That summation is a stateful operation that produces a KTable.
Understanding the Concepts
What is KStream?
KStream is the core abstraction in Kafka Streams for representing a stream of records. Each record is a key-value pair. Records arrive continuously from a Kafka topic, one at a time, in partition order.
The key concept is that KStream treats every record as an independent event. If you receive three records with key SAV2001, KStream sees three separate events. It does not replace or update earlier records.
You create a KStream by calling builder.stream("topic-name").
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class KStreamBasics {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Create a KStream from the bank-transactions topic
KStream<String, String> transactions = builder.stream("bank-transactions");
// Each record is an independent event
// Key: account ID (e.g., SAV2001)
// Value: transaction detail (e.g., DEPOSIT|50000|Mumbai)
transactions.peek((key, value) ->
System.out.println("Event: key=" + key + ", value=" + value));
transactions.to("transactions-log");
Topology topology = builder.build();
System.out.println("KStream topology:");
System.out.println(topology.describe());
}
}
Expected output:
KStream topology:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
--> KSTREAM-PEEK-0000000001
Processor: KSTREAM-PEEK-0000000001 (stores: [])
--> KSTREAM-SINK-0000000002
<-- KSTREAM-SOURCE-0000000000
Sink: KSTREAM-SINK-0000000002 (topic: transactions-log)
<-- KSTREAM-PEEK-0000000001
How does KStream handle multiple records with the same key?
KStream does not merge, update, or deduplicate records. Every record is processed independently regardless of its key. This is the fundamental difference between KStream and KTable.
If three deposit events arrive for account SAV2001, the KStream processes all three as separate events.
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class KStreamSameKeyDemo {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
builder.<String, String>stream("bank-transactions")
.mapValues(value -> "PROCESSED: " + value)
.to("processed-transactions");
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(builder.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"bank-transactions",
new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"processed-transactions",
new StringDeserializer(), new StringDeserializer());
// Three events with the SAME key: SAV2001
input.pipeInput("SAV2001", "DEPOSIT|50000|Mumbai|10:00");
input.pipeInput("SAV2001", "DEPOSIT|30000|Mumbai|11:00");
input.pipeInput("SAV2001", "WITHDRAWAL|10000|Delhi|12:00");
// KStream processes ALL THREE, not just the latest
System.out.println("Records in output: " + output.getQueueSize());
System.out.println("Record 1: " + output.readValue());
System.out.println("Record 2: " + output.readValue());
System.out.println("Record 3: " + output.readValue());
System.out.println();
System.out.println("KStream behavior: all 3 records processed");
System.out.println(" No replacement, no dedup, no merge");
System.out.println(" Each record is an independent event");
}
}
}
Expected output:
Records in output: 3
Record 1: PROCESSED: DEPOSIT|50000|Mumbai|10:00
Record 2: PROCESSED: DEPOSIT|30000|Mumbai|11:00
Record 3: PROCESSED: WITHDRAWAL|10000|Delhi|12:00
KStream behavior: all 3 records processed
No replacement, no dedup, no merge
Each record is an independent event
What stateless operations does KStream support?
Stateless operations process each record independently without remembering any previous records. KStream provides these stateless operations:
| Operation | What it does | Changes key? | Repartition? |
|---|---|---|---|
| filter(predicate) | Keep records matching condition | No | No |
| filterNot(predicate) | Remove records matching condition | No | No |
| map(mapper) | Transform both key and value | Yes | Yes (if key changes) |
| mapValues(mapper) | Transform value only | No | No |
| flatMap(mapper) | One record produces zero or more records (key+value) | Yes | Yes |
| flatMapValues(mapper) | One value produces zero or more values | No | No |
| selectKey(mapper) | Change the key only | Yes | Yes |
| peek(action) | Observe each record (for logging), pass through unchanged | No | No |
| branch(predicates) | Split into multiple streams by condition | No | No |
| merge(other) | Combine two streams into one | No | No |
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
public class KStreamStatelessOps {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> txns = builder.stream("bank-transactions");
// filter: keep only deposits
KStream<String, String> deposits = txns
.filter((key, value) -> value.startsWith("DEPOSIT"));
// mapValues: add a processing tag
KStream<String, String> tagged = deposits
.mapValues(value -> "VERIFIED|" + value);
// peek: log each record for monitoring
KStream<String, String> logged = tagged
.peek((key, value) ->
System.out.println("Verified deposit: " + key + " -> " + value));
// flatMapValues: split multi-item transactions into individual records
KStream<String, String> expanded = txns
.flatMapValues(value -> {
String[] items = value.split(";");
return java.util.Arrays.asList(items);
});
logged.to("verified-deposits");
expanded.to("expanded-transactions");
System.out.println("Stateless operations used:");
System.out.println(" filter() -> keep deposits only");
System.out.println(" mapValues() -> add VERIFIED tag");
System.out.println(" peek() -> log for monitoring");
System.out.println(" flatMapValues() -> split multi-item records");
}
}
Expected output:
Stateless operations used:
filter() -> keep deposits only
mapValues() -> add VERIFIED tag
peek() -> log for monitoring
flatMapValues() -> split multi-item records
How does KStream transition to stateful processing?
KStream itself is stateless. To perform stateful operations like counting, summing, or aggregating, you first group records by key using groupByKey() or groupBy(). This produces a KGroupedStream. Then you call count(), reduce(), or aggregate() to produce a KTable.
The KTable holds the latest aggregated result per key. As new records arrive on the KStream, the KTable updates.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KGroupedStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.streams.state.ValueAndTimestamp;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class KStreamToStateful {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Step 1: groupByKey produces KGroupedStream
KGroupedStream<String, String> grouped = transactions.groupByKey();
// Step 2: count() produces KTable<String, Long>
KTable<String, Long> txnCounts = grouped.count(
Materialized.as("transaction-counts"));
// The KTable holds the latest count per account
txnCounts.toStream().peek((key, count) ->
System.out.println("Account " + key + " has " + count + " transactions"))
.to("transaction-count-output");
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(builder.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"bank-transactions",
new StringSerializer(), new StringSerializer());
// SAV2001: 3 transactions
input.pipeInput("SAV2001", "DEPOSIT|50000");
input.pipeInput("SAV2001", "DEPOSIT|30000");
input.pipeInput("SAV2001", "WITHDRAWAL|10000");
// SAV2002: 1 transaction
input.pipeInput("SAV2002", "DEPOSIT|25000");
// Read state store directly
KeyValueStore<String, ValueAndTimestamp<Long>> store =
driver.getTimestampedKeyValueStore("transaction-counts");
System.out.println();
System.out.println("State store contents:");
System.out.println(" SAV2001: " + store.get("SAV2001").value() + " transactions");
System.out.println(" SAV2002: " + store.get("SAV2002").value() + " transaction");
System.out.println();
System.out.println("KStream (events) -> groupByKey -> count -> KTable (state)");
}
}
}
Expected output:
Account SAV2001 has 1 transactions
Account SAV2001 has 2 transactions
Account SAV2001 has 3 transactions
Account SAV2002 has 1 transactions
State store contents:
SAV2001: 3 transactions
SAV2002: 1 transaction
KStream (events) -> groupByKey -> count -> KTable (state)
Notice that the KTable emits an updated count after every input record. The state store holds the final accumulated value.
What is a complete banking project using KStream?
This project builds a payment notification service for National Bank of India. It reads payment events from a KStream, classifies them by type (UPI, NEFT, RTGS), enriches each with a processing fee, and routes notifications to type-specific topics.
Project structure
payment-notification-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── PaymentNotificationApp.java
│ ├── PaymentNotificationTopology.java
│ └── PaymentEvent.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── PaymentNotificationTest.java
PaymentEvent.java
package com.bank.streams;
import java.math.BigDecimal;
public class PaymentEvent {
private String paymentId;
private String fromAccount;
private String toAccount;
private BigDecimal amount;
private String type; // UPI, NEFT, RTGS
private String timestamp;
public PaymentEvent() {}
public PaymentEvent(String paymentId, String fromAccount, String toAccount,
BigDecimal amount, String type, String timestamp) {
this.paymentId = paymentId;
this.fromAccount = fromAccount;
this.toAccount = toAccount;
this.amount = amount;
this.type = type;
this.timestamp = timestamp;
}
public String getPaymentId() { return paymentId; }
public String getFromAccount() { return fromAccount; }
public String getToAccount() { return toAccount; }
public BigDecimal getAmount() { return amount; }
public String getType() { return type; }
public String getTimestamp() { return timestamp; }
@Override
public String toString() {
return paymentId + "|" + fromAccount + "|" + toAccount + "|"
+ amount + "|" + type + "|" + timestamp;
}
public static PaymentEvent fromString(String value) {
String[] p = value.split("\\|");
return new PaymentEvent(p[0], p[1], p[2],
new BigDecimal(p[3]), p[4], p[5]);
}
}
PaymentNotificationTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import java.math.BigDecimal;
import java.math.RoundingMode;
public class PaymentNotificationTopology {
// Fee schedule: UPI free, NEFT 2.50, RTGS 25.00
private static final BigDecimal NEFT_FEE = new BigDecimal("2.50");
private static final BigDecimal RTGS_FEE = new BigDecimal("25.00");
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> payments = builder.stream("payment-events");
// Enrich with processing fee
KStream<String, String> enriched = payments.mapValues(value -> {
PaymentEvent evt = PaymentEvent.fromString(value);
BigDecimal fee;
switch (evt.getType()) {
case "RTGS": fee = RTGS_FEE; break;
case "NEFT": fee = NEFT_FEE; break;
default: fee = BigDecimal.ZERO; break;
}
BigDecimal total = evt.getAmount().add(fee)
.setScale(2, RoundingMode.HALF_UP);
return evt.toString() + "|FEE:" + fee + "|TOTAL:" + total;
});
// Route by payment type to separate notification topics
KStream<String, String> upi = enriched
.filter((key, value) -> value.contains("|UPI|"));
KStream<String, String> neft = enriched
.filter((key, value) -> value.contains("|NEFT|"));
KStream<String, String> rtgs = enriched
.filter((key, value) -> value.contains("|RTGS|"));
upi.to("notifications-upi");
neft.to("notifications-neft");
rtgs.to("notifications-rtgs");
return builder.build();
}
}
PaymentNotificationApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class PaymentNotificationApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"payment-notification-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
Topology topology = PaymentNotificationTopology.build();
KafkaStreams streams = new KafkaStreams(topology, props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Payment notification service stopped");
}));
try {
streams.start();
System.out.println("Payment notification service started");
System.out.println(" Input: payment-events");
System.out.println(" Output: notifications-upi, notifications-neft, notifications-rtgs");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
PaymentNotificationTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class PaymentNotificationTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(PaymentNotificationTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"payment-events",
new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> upiOut = driver.createOutputTopic(
"notifications-upi",
new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> neftOut = driver.createOutputTopic(
"notifications-neft",
new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> rtgsOut = driver.createOutputTopic(
"notifications-rtgs",
new StringDeserializer(), new StringDeserializer());
// UPI payment: Rajesh pays Priya 500 rupees, no fee
input.pipeInput("PAY001",
"PAY001|SAV2001|SAV2002|500|UPI|2026-11-13T10:00:00");
// NEFT payment: 75,000 rupees, fee 2.50
input.pipeInput("PAY002",
"PAY002|SAV2001|CUR3001|75000|NEFT|2026-11-13T10:15:00");
// RTGS payment: 5,00,000 rupees, fee 25.00
input.pipeInput("PAY003",
"PAY003|CUR3001|SAV2003|500000|RTGS|2026-11-13T10:30:00");
System.out.println("UPI notifications: " + upiOut.getQueueSize());
System.out.println(" " + upiOut.readValue());
System.out.println();
System.out.println("NEFT notifications: " + neftOut.getQueueSize());
System.out.println(" " + neftOut.readValue());
System.out.println();
System.out.println("RTGS notifications: " + rtgsOut.getQueueSize());
System.out.println(" " + rtgsOut.readValue());
System.out.println();
System.out.println("Test passed: 3 payments routed to correct type topics with fees");
}
}
}
Expected output:
UPI notifications: 1
PAY001|SAV2001|SAV2002|500|UPI|2026-11-13T10:00:00|FEE:0|TOTAL:500.00
NEFT notifications: 1
PAY002|SAV2001|CUR3001|75000|NEFT|2026-11-13T10:15:00|FEE:2.50|TOTAL:75002.50
RTGS notifications: 1
PAY003|CUR3001|SAV2003|500000|RTGS|2026-11-13T10:30:00|FEE:25.00|TOTAL:500025.00
Test passed: 3 payments routed to correct type topics with fees
How to run
# Step 1: Create topics
kafka-topics.sh --create --topic payment-events --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic notifications-upi --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic notifications-neft --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic notifications-rtgs --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
# Step 2: Build
cd payment-notification-service
mvn clean package
# Step 3: Run test (no broker needed)
java -cp target/payment-notification-service-1.0.0.jar com.bank.streams.PaymentNotificationTest
# Step 4: Run app
java -cp target/payment-notification-service-1.0.0.jar com.bank.streams.PaymentNotificationApp
# Step 5: Produce
kafka-console-producer.sh --topic payment-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# PAY001:PAY001|SAV2001|SAV2002|500|UPI|2026-11-13T10:00:00
# PAY002:PAY002|SAV2001|CUR3001|75000|NEFT|2026-11-13T10:15:00
# PAY003:PAY003|CUR3001|SAV2003|500000|RTGS|2026-11-13T10:30:00
# Step 6: Consume each output topic
kafka-console-consumer.sh --topic notifications-upi --from-beginning --bootstrap-server localhost:9092
kafka-console-consumer.sh --topic notifications-neft --from-beginning --bootstrap-server localhost:9092
kafka-console-consumer.sh --topic notifications-rtgs --from-beginning --bootstrap-server localhost:9092
What are common mistakes when working with KStream?
Wrong: treating KStream as a table (expecting latest-value semantics)
package com.bank.streams;
public class WrongKStreamAsTable {
public static void main(String[] args) {
System.out.println("WRONG: Expecting KStream to keep only the latest value per key");
System.out.println();
System.out.println(" Input (3 records, same key SAV2001):");
System.out.println(" SAV2001 -> DEPOSIT|50000");
System.out.println(" SAV2001 -> DEPOSIT|30000");
System.out.println(" SAV2001 -> WITHDRAWAL|10000");
System.out.println();
System.out.println(" Developer expects: only WITHDRAWAL|10000 in output");
System.out.println(" Actual result: ALL THREE records in output");
System.out.println();
System.out.println(" KStream is an event log, not a key-value store");
System.out.println(" Every record is processed independently");
System.out.println();
System.out.println(" Fix: if you need latest-value-per-key, use KTable");
System.out.println(" builder.table('topic') instead of builder.stream('topic')");
}
}
Expected output:
WRONG: Expecting KStream to keep only the latest value per key
Input (3 records, same key SAV2001):
SAV2001 -> DEPOSIT|50000
SAV2001 -> DEPOSIT|30000
SAV2001 -> WITHDRAWAL|10000
Developer expects: only WITHDRAWAL|10000 in output
Actual result: ALL THREE records in output
KStream is an event log, not a key-value store
Every record is processed independently
Fix: if you need latest-value-per-key, use KTable
builder.table('topic') instead of builder.stream('topic')
Correct: use KStream for events, KTable for state
package com.bank.streams;
public class CorrectKStreamVsKTable {
public static void main(String[] args) {
System.out.println("CORRECT: Choose the right abstraction");
System.out.println();
System.out.println(" Use KStream (builder.stream) when:");
System.out.println(" - Every event matters: transactions, payments, logins");
System.out.println(" - You process each record independently");
System.out.println(" - Duplicate keys mean multiple events");
System.out.println();
System.out.println(" Use KTable (builder.table) when:");
System.out.println(" - You need the latest value per key: balance, status, profile");
System.out.println(" - A new record with the same key replaces the old one");
System.out.println(" - You need lookup/reference data for joins");
}
}
Expected output:
CORRECT: Choose the right abstraction
Use KStream (builder.stream) when:
- Every event matters: transactions, payments, logins
- You process each record independently
- Duplicate keys mean multiple events
Use KTable (builder.table) when:
- You need the latest value per key: balance, status, profile
- A new record with the same key replaces the old one
- You need lookup/reference data for joins
What should a senior or principal engineer know?
At the senior level, you should understand that KStream is append-only and every record is independent. You should know when to use KStream versus KTable, and how groupByKey transitions a KStream into stateful processing.
At the principal level, consider:
- Event time vs processing time: KStream records carry a timestamp. By default, Kafka Streams uses the record's event time (when the producer created it). You can override this with a custom TimestampExtractor.
- Out-of-order records: within a partition, records are ordered. Across partitions, order is not guaranteed. Design your KStream processing to handle late or out-of-order events, especially for windowed operations.
- Repartitioning cost: operations that change the key (map, selectKey, flatMap) cause repartitioning. Kafka Streams writes records to an internal repartition topic and reads them back, doubling the I/O. Use mapValues or flatMapValues when you do not need to change the key.
- Serialization: KStream uses Serdes (serializer-deserializer pairs) for keys and values. Mismatched serdes cause deserialization exceptions at runtime. Set default serdes globally or override per operation.
- Backpressure: KStream does not have built-in backpressure. If the downstream sink topic is slow, records accumulate in the processing buffer. Monitor
records-lag-maxandprocess-ratemetrics.
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;
public class KStreamPrincipalKnowledge {
public static void main(String[] args) {
System.out.println("Principal-level KStream knowledge:");
System.out.println();
System.out.println("1. Event time vs processing time:");
System.out.println(" Default: event time from producer timestamp");
System.out.println(" Custom: implement TimestampExtractor to parse from value");
System.out.println();
System.out.println("2. Repartitioning cost:");
System.out.println(" map() / selectKey() / flatMap() -> repartition topic -> 2x I/O");
System.out.println(" mapValues() / flatMapValues() -> no repartition -> fast");
System.out.println();
System.out.println("3. Serialization:");
System.out.println(" Mismatch = SerializationException at runtime");
System.out.println(" Always set default.key.serde and default.value.serde");
System.out.println();
System.out.println("4. Monitoring KStream throughput:");
System.out.println(" records-lag-max: how far behind the consumer is");
System.out.println(" process-rate: records processed per second");
System.out.println(" process-latency: time per record in the topology");
}
}
Expected output:
Principal-level KStream knowledge:
1. Event time vs processing time:
Default: event time from producer timestamp
Custom: implement TimestampExtractor to parse from value
2. Repartitioning cost:
map() / selectKey() / flatMap() -> repartition topic -> 2x I/O
mapValues() / flatMapValues() -> no repartition -> fast
3. Serialization:
Mismatch = SerializationException at runtime
Always set default.key.serde and default.value.serde
4. Monitoring KStream throughput:
records-lag-max: how far behind the consumer is
process-rate: records processed per second
process-latency: time per record in the topology
What are common interview follow-up questions?
- Is KStream bounded or unbounded? (Unbounded. Records keep arriving as long as producers publish to the topic. There is no end-of-stream marker.)
- Can KStream handle deletes? (Not directly. A null value is a tombstone and is typically filtered out by KStream. KTable handles tombstones by removing the key from the state store.)
- What is the difference between KStream and Java Streams? (KStream is for distributed, real-time processing of Kafka records. Java Streams is for in-memory processing of collections. They share similar method names like filter and map, but are completely different abstractions.)
- Can you read from multiple topics into one KStream? (Yes. Pass a list or a regex pattern:
builder.stream(Arrays.asList("topic1", "topic2"))orbuilder.stream(Pattern.compile("bank-.*")).) - What happens if the producer changes the value format? (Deserialization fails. Use a DeserializationExceptionHandler to log and skip bad records instead of crashing the stream thread.)
- Can a KStream write to multiple output topics? (Yes. Branch the stream and call .to() on each branch. Or use
.to(TopicNameExtractor)to dynamically choose the topic per record.)
Visual Overview
KStream: Unbounded Event Stream
=================================
Producer publishes continuously:
10:00 SAV2001 -> DEPOSIT|50000
10:01 SAV2002 -> DEPOSIT|25000
10:02 SAV2001 -> DEPOSIT|30000
10:03 SAV2001 -> WITHDRAWAL|10000
10:04 CUR3001 -> NEFT|75000
... (never ends)
KStream sees EVERY record:
┌──────────────────────────┐
│ KStream │
│ │
│ SAV2001 DEPOSIT 50000 │ event 1
│ SAV2002 DEPOSIT 25000 │ event 2
│ SAV2001 DEPOSIT 30000 │ event 3
│ SAV2001 WITHDRAWAL 10000 │ event 4
│ CUR3001 NEFT 75000 │ event 5
│ ... │ unbounded
└──────────────────────────┘
3 records for SAV2001 = 3 separate events
(not 1 updated record)
KStream Stateless vs Stateful
==============================
KStream ──> filter/map/peek ──> KStream (stateless)
KStream ──> groupByKey ──> KGroupedStream
│
├──> count() ──> KTable
├──> reduce() ──> KTable
└──> aggregate() ──> KTable
Key Takeaway
KStream is the fundamental abstraction in Kafka Streams for representing a continuous, unbounded flow of events from a Kafka topic. Every record is treated as an independent event. Three deposits for account SAV2001 mean three separate events, not one updated balance. This is the core difference from KTable, which keeps only the latest value per key.
Stateless operations on KStream like filter, mapValues, flatMapValues, and peek process each record independently with no memory of previous records. When you need to aggregate, count, or reduce, you call groupByKey() to get a KGroupedStream, then count(), reduce(), or aggregate() to produce a KTable with accumulated state.
In our National Bank of India payment notification project, we read payment events as a KStream, enriched each with a processing fee based on type (UPI free, NEFT 2.50, RTGS 25.00), and routed notifications to type-specific topics. The unit test verified all three payment types with correct fees and totals. A junior developer can clone the project, run the test without a broker, and see the exact output.
At the principal level, remember that repartitioning happens when you change the key, doubling I/O. Use mapValues and flatMapValues when you do not need to change the key. Monitor records-lag-max and process-rate to detect throughput problems early.
Remember one sentence: KStream is an unbounded log of independent events where every record is processed as a new fact, not an update to a previous value.
---