Definition
- Stateful operation is a stream processing step that maintains a local state store to accumulate information across multiple records with the same key.
- groupByKey() groups records by their current key without changing it, which avoids repartitioning.
- groupBy() groups records by a new key computed from the record, which triggers repartitioning because the key changes.
- count() is an aggregation that counts the number of records per grouped key and stores the count in a state store.
- reduce() is an aggregation that combines two values into one using a binary function, producing one accumulated result per key.
- aggregate() is the most general aggregation: it takes an initializer for new keys, an adder function, and a Materialized store configuration.
- Changelog topic is an internal Kafka topic that backs each state store for fault-tolerant recovery after a crash or rebalance.
Think of the daily reporting desk at National Bank of India. Each clerk at the desk keeps a register (state store) to track running totals. One clerk counts the number of transactions per account (count). Another clerk tracks the largest single transaction per account (reduce). A third clerk maintains a running balance per account, adding deposits and subtracting withdrawals (aggregate). If a clerk's register is destroyed, they rebuild it from the carbon copies stored in the filing cabinet (changelog topic). The groupByKey step is like sorting slips by account number before handing them to the clerks.
Understanding the Concepts
What are stateful operations in Kafka Streams?
Stateful operations process records while maintaining local state that persists across multiple records. Unlike stateless operations that forget everything after processing a record, stateful operations remember accumulated results in a state store.
Every stateful operation creates two things: a local state store (RocksDB by default) and a changelog topic in Kafka. The state store holds the current aggregated values. The changelog topic is a backup: if the application crashes or a partition rebalances to another instance, the state store is rebuilt from the changelog.
package com.bank.streams;
public class StatefulOperationsOverview {
public static void main(String[] args) {
System.out.println("Stateful operations in Kafka Streams:");
System.out.println(" groupByKey -> count : transactions per account");
System.out.println(" groupByKey -> reduce : max transaction per account");
System.out.println(" groupByKey -> aggregate: running balance per account");
System.out.println(" groupBy -> count : transactions per branch");
System.out.println();
System.out.println("Each creates:");
System.out.println(" 1. Local state store (RocksDB)");
System.out.println(" 2. Changelog topic (for recovery)");
}
}
Expected output:
Stateful operations in Kafka Streams:
groupByKey -> count : transactions per account
groupByKey -> reduce : max transaction per account
groupByKey -> aggregate: running balance per account
groupBy -> count : transactions per branch
Each creates:
1. Local state store (RocksDB)
2. Changelog topic (for recovery)
How does groupByKey work?
groupByKey() groups records by their existing key. It returns a KGroupedStream, which you can then pass to count(), reduce(), or aggregate(). Because the key does not change, no repartitioning is needed.
At National Bank of India, transactions arrive keyed by account ID (SAV2001, CUR3001). Rajesh calls groupByKey() to group all transactions by account before counting or aggregating them.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KGroupedStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.Topology;
public class GroupByKeyExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Group by existing key (account ID). No repartition needed.
KGroupedStream<String, String> grouped = transactions.groupByKey();
// Count transactions per account
KTable<String, Long> txnCounts = grouped.count();
txnCounts.toStream().foreach((accountId, count) ->
System.out.println("Account " + accountId + ": " + count + " transactions"));
Topology topology = builder.build();
System.out.println("groupByKey topology (no repartition):");
System.out.println(topology.describe());
}
}
Expected output:
Account SAV2001: 1 transactions
Account SAV2001: 2 transactions
Account CUR3001: 1 transactions
Account SAV2001: 3 transactions
groupByKey topology (no repartition):
No repartition topic in the topology description.
How does groupBy work?
groupBy() groups records by a new key computed from the record. Because the key changes, the resulting stream is repartitioned through an internal topic. Use groupBy() when you need to aggregate by a different key than the record's current key.
At National Bank of India, transactions arrive keyed by account ID. Rajesh needs to count transactions per branch. The branch code is the first three characters of the account ID (SAV, CUR). He uses groupBy() to rekey by branch code.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KGroupedStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
public class GroupByExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Group by branch code (new key). Triggers repartitioning.
KGroupedStream<String, String> byBranch = transactions.groupBy(
(accountId, value) -> accountId.substring(0, 3),
Grouped.with(Serdes.String(), Serdes.String())
);
// Count transactions per branch
KTable<String, Long> branchCounts = byBranch.count();
branchCounts.toStream().foreach((branch, count) ->
System.out.println("Branch " + branch + ": " + count + " transactions"));
Topology topology = builder.build();
System.out.println("groupBy topology (with repartition):");
System.out.println(topology.describe());
}
}
Expected output:
Branch SAV: 1 transactions
Branch SAV: 2 transactions
Branch CUR: 1 transactions
Branch SAV: 3 transactions
groupBy topology (with repartition):
Contains internal repartition topic in topology description.
How do groupByKey and groupBy compare?
The key difference is whether the grouping key changes. groupByKey uses the existing key; groupBy computes a new key.
| Feature | groupByKey | groupBy |
|---|---|---|
| Key change | No, uses existing key | Yes, computes new key |
| Repartitioning | No | Yes, creates internal repartition topic |
| Performance | Faster, no network overhead | Slower, records written to repartition topic |
| When to use | Aggregate by the same key | Aggregate by a different key |
| Banking example | Count per account (key=account) | Count per branch (key=branch code) |
| Serde requirement | Uses default serdes | Must specify serdes via Grouped.with() |
package com.bank.streams;
public class GroupByKeyVsGroupBy {
public static void main(String[] args) {
System.out.println("=== groupByKey ===");
System.out.println(" Input key: SAV2001 (account ID)");
System.out.println(" Group key: SAV2001 (same, no change)");
System.out.println(" Repartition: NO");
System.out.println(" Use case: count transactions per account");
System.out.println();
System.out.println("=== groupBy ===");
System.out.println(" Input key: SAV2001 (account ID)");
System.out.println(" Group key: SAV (branch code, new key)");
System.out.println(" Repartition: YES");
System.out.println(" Use case: count transactions per branch");
System.out.println();
System.out.println("Rule: Use groupByKey when possible. Use groupBy only");
System.out.println("when you need to change the grouping key.");
}
}
Expected output:
=== groupByKey ===
Input key: SAV2001 (account ID)
Group key: SAV2001 (same, no change)
Repartition: NO
Use case: count transactions per account
=== groupBy ===
Input key: SAV2001 (account ID)
Group key: SAV (branch code, new key)
Repartition: YES
Use case: count transactions per branch
Rule: Use groupByKey when possible. Use groupBy only
when you need to change the grouping key.
How does count work?
count() counts the number of records per key in a grouped stream. It returns a KTable
At National Bank of India, Rajesh counts the number of transactions per account per day to detect accounts with unusually high activity.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
public class CountExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Count transactions per account
KTable<String, Long> txnCounts = transactions
.groupByKey()
.count(Materialized.as("txn-count-store"));
// Emit count changes
txnCounts.toStream().foreach((accountId, count) ->
System.out.println(accountId + ": " + count + " transactions"));
Topology topology = builder.build();
System.out.println("Count topology with named state store: txn-count-store");
}
}
Expected output (records arrive one by one):
SAV2001: 1 transactions
SAV2001: 2 transactions
CUR3001: 1 transactions
SAV2001: 3 transactions
Count topology with named state store: txn-count-store
How does reduce work?
reduce() combines records with the same key using a binary function. The function takes two values and returns one. It runs repeatedly: the first record becomes the initial value, and each subsequent record is combined with the accumulated result.
At National Bank of India, Rajesh uses reduce to find the largest transaction per account. The reduce function compares two transaction amounts and keeps the larger one.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
public class ReduceExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Reduce to find max transaction amount per account
KTable<String, String> maxPerAccount = transactions
.groupByKey()
.reduce(
(currentMax, newValue) -> {
BigDecimal current = new BigDecimal(currentMax.split("\\|")[2]);
BigDecimal incoming = new BigDecimal(newValue.split("\\|")[2]);
return incoming.compareTo(current) > 0 ? newValue : currentMax;
},
Materialized.as("max-txn-store")
);
maxPerAccount.toStream().foreach((accountId, maxTxn) ->
System.out.println("Max for " + accountId + ": " + maxTxn));
Topology topology = builder.build();
System.out.println("Reduce topology: find max transaction per account");
}
}
Expected output:
Max for SAV2001: SAV2001|DEPOSIT|50000|salary credit
Max for SAV2001: SAV2001|NEFT|250000|property payment
Max for CUR3001: CUR3001|RTGS|500000|vendor payment
Max for SAV2001: SAV2001|NEFT|250000|property payment
Reduce topology: find max transaction per account
After 3 records for SAV2001 (50000, 250000, 10000):
Max stays at 250000 because 10000 < 250000.
How does aggregate work?
aggregate() is the most flexible stateful operation. It takes three parameters: an initializer that creates the starting value for a new key, an adder function that combines the current aggregate with a new record, and a Materialized store configuration.
Unlike reduce(), aggregate() can produce a result of a different type than the input. The initializer runs once per new key. The adder runs for every subsequent record.
At National Bank of India, Rajesh uses aggregate to maintain a running balance per account. Deposits are added and withdrawals are subtracted.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.math.BigDecimal;
public class AggregateExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Aggregate running balance per account
KTable<String, String> balances = transactions
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.aggregate(
// Initializer: starting balance for new accounts
() -> "0",
// Adder: update balance based on transaction type
(accountId, txnValue, currentBalance) -> {
String[] parts = txnValue.split("\\|");
String type = parts[1];
BigDecimal amount = new BigDecimal(parts[2]);
BigDecimal balance = new BigDecimal(currentBalance);
if ("DEPOSIT".equals(type)) {
balance = balance.add(amount);
} else if ("WITHDRAW".equals(type)) {
balance = balance.subtract(amount);
}
return balance.toPlainString();
},
// Store configuration
Materialized.as("balance-store")
);
balances.toStream().foreach((accountId, balance) ->
System.out.println("Balance " + accountId + ": " + balance + " INR"));
Topology topology = builder.build();
System.out.println("Aggregate topology: running balance per account");
}
}
Expected output:
Balance SAV2001: 50000 INR (DEPOSIT 50000)
Balance SAV2001: 40000 INR (WITHDRAW 10000)
Balance SAV2001: 65000 INR (DEPOSIT 25000)
Balance CUR3001: 500000 INR (DEPOSIT 500000)
Aggregate topology: running balance per account
What are windowed aggregations?
Windowed aggregations group records by key and time window. Instead of aggregating all records for a key forever, you aggregate only records that fall within a specific time range.
- Kafka Streams supports three window types:
- Tumbling window: fixed-size, non-overlapping (e.g., every 1 hour)
- Hopping window: fixed-size, overlapping (e.g., 1-hour window every 15 minutes)
- Session window: variable-size, based on activity gaps (e.g., group events with gaps less than 30 minutes)
At National Bank of India, Rajesh counts transactions per account per hour to detect suspicious burst activity.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class WindowedCountExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Count transactions per account per 1-hour tumbling window
KTable<Windowed<String>, Long> hourlyCounts = transactions
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
.count(Materialized.as("hourly-count-store"));
hourlyCounts.toStream().foreach((windowedKey, count) -> {
String accountId = windowedKey.key();
long windowStart = windowedKey.window().start();
long windowEnd = windowedKey.window().end();
System.out.println(accountId + " [" + windowStart + "-" + windowEnd + "]: "
+ count + " transactions");
});
Topology topology = builder.build();
System.out.println("Windowed count: transactions per account per hour");
}
}
Expected output:
SAV2001 [1700000000000-1700003600000]: 1 transactions
SAV2001 [1700000000000-1700003600000]: 2 transactions
SAV2001 [1700000000000-1700003600000]: 3 transactions
CUR3001 [1700000000000-1700003600000]: 1 transactions
Windowed count: transactions per account per hour
All records fell within the same 1-hour window.
If a record arrives in the next hour, a new window starts.
What is a complete banking project using stateful operations?
This project is a Daily Transaction Analytics Service for National Bank of India. It reads transactions and produces three outputs: transaction count per account, maximum transaction per account, and running balance per account.
Project structure
txn-analytics-service/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── TxnAnalyticsApp.java
├── TxnAnalyticsTopology.java
├── Transaction.java
└── TxnAnalyticsTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>txn-analytics-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
Transaction.java
package com.bank.streams;
import java.math.BigDecimal;
public class Transaction {
private String accountId;
private String type;
private BigDecimal amount;
private String description;
public Transaction() {}
public Transaction(String accountId, String type,
BigDecimal amount, String description) {
this.accountId = accountId;
this.type = type;
this.amount = amount;
this.description = description;
}
public String getAccountId() { return accountId; }
public String getType() { return type; }
public BigDecimal getAmount() { return amount; }
public String getDescription() { return description; }
@Override
public String toString() {
return accountId + "|" + type + "|" + amount + "|" + description;
}
public static Transaction fromString(String value) {
String[] parts = value.split("\\|");
return new Transaction(
parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
TxnAnalyticsTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.math.BigDecimal;
public class TxnAnalyticsTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// 1. Count transactions per account
KTable<String, Long> txnCounts = transactions
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.count(Materialized.as("txn-count-store"));
txnCounts.toStream()
.mapValues(count -> String.valueOf(count))
.to("txn-counts");
// 2. Reduce to find max transaction per account
KTable<String, String> maxTxn = transactions
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.reduce(
(current, incoming) -> {
BigDecimal currentAmt = new BigDecimal(current.split("\\|")[2]);
BigDecimal incomingAmt = new BigDecimal(incoming.split("\\|")[2]);
return incomingAmt.compareTo(currentAmt) > 0 ? incoming : current;
},
Materialized.as("max-txn-store")
);
maxTxn.toStream().to("max-transactions");
// 3. Aggregate running balance per account
KTable<String, String> balances = transactions
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.aggregate(
() -> "0",
(accountId, txnValue, currentBalance) -> {
Transaction txn = Transaction.fromString(txnValue);
BigDecimal balance = new BigDecimal(currentBalance);
if ("DEPOSIT".equals(txn.getType())) {
balance = balance.add(txn.getAmount());
} else if ("WITHDRAW".equals(txn.getType())) {
balance = balance.subtract(txn.getAmount());
}
return balance.toPlainString();
},
Materialized.as("balance-store")
);
balances.toStream().to("account-balances");
return builder.build();
}
}
TxnAnalyticsApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class TxnAnalyticsApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"txn-analytics-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
TxnAnalyticsTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
}));
try {
streams.start();
System.out.println("Transaction analytics service started");
System.out.println(" Input: bank-transactions");
System.out.println(" Output: txn-counts, max-transactions, account-balances");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
Expected output (on startup):
Transaction analytics service started
Input: bank-transactions
Output: txn-counts, max-transactions, account-balances
TxnAnalyticsTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.LongDeserializer;
import java.util.Properties;
public class TxnAnalyticsTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(TxnAnalyticsTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"bank-transactions", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> counts = driver.createOutputTopic(
"txn-counts", new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> maxTxns = driver.createOutputTopic(
"max-transactions", new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> balances = driver.createOutputTopic(
"account-balances", new StringDeserializer(), new StringDeserializer());
input.pipeInput("SAV2001", "SAV2001|DEPOSIT|50000|salary credit");
input.pipeInput("SAV2001", "SAV2001|NEFT|250000|property payment");
input.pipeInput("SAV2001", "SAV2001|WITHDRAW|10000|atm withdrawal");
input.pipeInput("CUR3001", "CUR3001|DEPOSIT|500000|vendor payment");
System.out.println("=== Transaction Counts ===");
while (!counts.isEmpty()) {
var r = counts.readKeyValue();
System.out.println(" " + r.key + ": " + r.value);
}
System.out.println("=== Max Transactions ===");
while (!maxTxns.isEmpty()) {
var r = maxTxns.readKeyValue();
System.out.println(" " + r.key + ": " + r.value);
}
System.out.println("=== Balances ===");
while (!balances.isEmpty()) {
var r = balances.readKeyValue();
System.out.println(" " + r.key + ": " + r.value + " INR");
}
}
}
}
Expected output:
=== Transaction Counts ===
SAV2001: 1
SAV2001: 2
SAV2001: 3
CUR3001: 1
=== Max Transactions ===
SAV2001: SAV2001|DEPOSIT|50000|salary credit
SAV2001: SAV2001|NEFT|250000|property payment
SAV2001: SAV2001|NEFT|250000|property payment
CUR3001: CUR3001|DEPOSIT|500000|vendor payment
=== Balances ===
SAV2001: 50000 INR
SAV2001: 300000 INR
SAV2001: 290000 INR
CUR3001: 500000 INR
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics
kafka-topics.sh --create --topic bank-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic txn-counts --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic max-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic account-balances --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build and run
cd txn-analytics-service
mvn clean package
java -cp target/txn-analytics-service-1.0.0.jar com.bank.streams.TxnAnalyticsApp
# Step 4: Produce test messages
kafka-console-producer.sh --topic bank-transactions --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|DEPOSIT|50000|salary credit
# SAV2001:SAV2001|NEFT|250000|property payment
# SAV2001:SAV2001|WITHDRAW|10000|atm withdrawal
# CUR3001:CUR3001|DEPOSIT|500000|vendor payment
What are common mistakes with stateful operations?
The most common mistake is using groupBy() when groupByKey() is sufficient, which triggers an unnecessary repartition.
Wrong: using groupBy with the same key
package com.bank.streams;
public class WrongGroupByWithSameKey {
public static void main(String[] args) {
System.out.println("WRONG: Using groupBy() when key does not change");
System.out.println();
System.out.println("Code:");
System.out.println(" transactions.groupBy(");
System.out.println(" (key, value) -> key,");
System.out.println(" Grouped.with(Serdes.String(), Serdes.String()))");
System.out.println(" .count();");
System.out.println();
System.out.println("Problem: groupBy creates a repartition topic even though");
System.out.println("the key is the same. This wastes network, storage, and time.");
}
}
Expected output:
WRONG: Using groupBy() when key does not change
Code:
transactions.groupBy(
(key, value) -> key,
Grouped.with(Serdes.String(), Serdes.String()))
.count();
Problem: groupBy creates a repartition topic even though
the key is the same. This wastes network, storage, and time.
Correct: using groupByKey when key stays the same
package com.bank.streams;
public class CorrectGroupByKey {
public static void main(String[] args) {
System.out.println("CORRECT: Using groupByKey() when key does not change");
System.out.println();
System.out.println("Code:");
System.out.println(" transactions.groupByKey()");
System.out.println(" .count();");
System.out.println();
System.out.println("No repartition topic created.");
System.out.println("Same result, better performance.");
}
}
Expected output:
CORRECT: Using groupByKey() when key does not change
Code:
transactions.groupByKey()
.count();
No repartition topic created.
Same result, better performance.
Wrong: not naming the state store
package com.bank.streams;
public class WrongUnnamedStore {
public static void main(String[] args) {
System.out.println("WRONG: Not naming the state store");
System.out.println();
System.out.println("Code:");
System.out.println(" transactions.groupByKey().count();");
System.out.println();
System.out.println("Problem: Kafka Streams assigns a generated name like");
System.out.println("KSTREAM-AGGREGATE-STATE-STORE-0000000003.");
System.out.println("If you add or remove operations, the generated name changes.");
System.out.println("This makes the old changelog topic incompatible.");
System.out.println("State cannot be restored from the old changelog.");
System.out.println();
System.out.println("CORRECT: Name the store explicitly");
System.out.println(" transactions.groupByKey().count(Materialized.as(\"txn-count-store\"));");
System.out.println("The changelog topic name is now stable across topology changes.");
}
}
Expected output:
WRONG: Not naming the state store
Code:
transactions.groupByKey().count();
Problem: Kafka Streams assigns a generated name like
KSTREAM-AGGREGATE-STATE-STORE-0000000003.
If you add or remove operations, the generated name changes.
This makes the old changelog topic incompatible.
State cannot be restored from the old changelog.
CORRECT: Name the store explicitly
transactions.groupByKey().count(Materialized.as("txn-count-store"));
The changelog topic name is now stable across topology changes.
What should a senior or principal engineer know?
At the senior level, you should understand how state stores and changelog topics affect your application's performance, storage, and recovery time.
- Changelog compaction: Changelog topics use cleanup.policy=compact by default. This keeps only the latest value per key, limiting storage growth. Never change this to delete, or state recovery after a crash will be incomplete.
- State store sizing: The state store holds one entry per unique key. If you aggregate 50 million accounts, the RocksDB store holds 50 million entries. Monitor the state store size via JMX metrics.
- Recovery time: After a rebalance, the state store is rebuilt from the changelog topic. Recovery time is proportional to the number of entries. Use standby replicas (num.standby.replicas=1) to keep a warm copy on another instance.
- Caching: Kafka Streams caches state store updates and flushes them periodically (cache.max.bytes.buffering). This reduces the number of records written to the changelog topic and emitted downstream. Set this to 10MB or more in production.
- Exactly-once with aggregations: When processing.guarantee=exactly_once_v2, each aggregate update is part of a Kafka transaction. This ensures that the state store update and the changelog write are atomic.
- Multiple aggregations: If you call groupByKey().count() and groupByKey().reduce() on the same stream, two separate state stores and two changelog topics are created. Each groupByKey() call creates an independent processing path.
package com.bank.streams;
public class SeniorStatefulKnowledge {
public static void main(String[] args) {
System.out.println("Senior checklist for stateful operations:");
System.out.println(" 1. Always name state stores with Materialized.as()");
System.out.println(" 2. Monitor state store size via JMX");
System.out.println(" 3. Set num.standby.replicas=1 for faster recovery");
System.out.println(" 4. Set cache.max.bytes.buffering >= 10MB in production");
System.out.println(" 5. Keep changelog compaction policy as compact");
System.out.println(" 6. Multiple aggregations = multiple stores + changelogs");
System.out.println(" 7. Use exactly_once_v2 for atomic state + changelog updates");
}
}
Expected output:
Senior checklist for stateful operations:
1. Always name state stores with Materialized.as()
2. Monitor state store size via JMX
3. Set num.standby.replicas=1 for faster recovery
4. Set cache.max.bytes.buffering >= 10MB in production
5. Keep changelog compaction policy as compact
6. Multiple aggregations = multiple stores + changelogs
7. Use exactly_once_v2 for atomic state + changelog updates
What are common interview follow-up questions?
- What is the difference between reduce and aggregate? (reduce combines two values of the same type; aggregate can produce a different type because it takes an initializer. Use reduce for simple combinations like max or sum; use aggregate when the result type differs from the input.)
- Can you do a stateful operation without groupByKey or groupBy? (No. Stateful operations like count, reduce, and aggregate require a grouped stream. You must call groupByKey() or groupBy() first.)
- What happens if a record has a null key? (Records with null keys are dropped by groupByKey() and groupBy(). They cannot be grouped because there is no key to group by. Always filter out null keys before grouping.)
- How do windowed aggregations handle late-arriving records? (By default, late records outside the window's grace period are dropped. Use .windowedBy(TimeWindows.ofSizeAndGrace()) to set a grace period. Records arriving within the grace period update the window.)
- Can you query the state store while the app is running? (Yes, using Interactive Queries. Call streams.store(StoreQueryParameters.fromNameAndType()) to get a read-only view of the store.)
- What is the difference between count on KGroupedStream and KGroupedTable? (KGroupedStream.count() increments for every record. KGroupedTable.count() adjusts: it subtracts for the old value and adds for the new value when a key is updated.)
Visual Overview
Stateful Operations Flow
==========================
┌───────────────────────────┐
│ bank-transactions │
│ SAV2001|DEPOSIT|50000 │
│ SAV2001|NEFT|250000 │
│ SAV2001|WITHDRAW|10000 │
│ CUR3001|DEPOSIT|500000 │
└────────────┬──────────────┘
│
▼
┌───────────────────────────┐
│ groupByKey() │
│ (no repartition) │
└──────┬──────┬──────┬──────┘
│ │ │
▼ ▼ ▼
┌────────┐┌────────┐┌──────────┐
│ count()││reduce()││aggregate()│
└───┬────┘└───┬────┘└─────┬────┘
│ │ │
▼ ▼ ▼
┌────────┐┌────────┐┌──────────┐
│ State ││ State ││ State │
│ Store ││ Store ││ Store │
│ (count)││ (max) ││ (balance)│
└───┬────┘└───┬────┘└─────┬────┘
│ │ │
▼ ▼ ▼
┌────────┐┌────────┐┌──────────┐
│txn- ││max- ││account- │
│counts ││txns ││balances │
└────────┘└────────┘└──────────┘
State Store + Changelog
=========================
┌──────────────────────┐
│ Aggregation Operation │
└──────────┬───────────┘
│ write
▼
┌──────────────────────┐ ┌───────────────────────┐
│ Local State Store │────→│ Changelog Topic │
│ (RocksDB) │ │ (compact, in Kafka) │
│ SAV2001: 290000 │ │ SAV2001: 290000 │
│ CUR3001: 500000 │ │ CUR3001: 500000 │
└──────────────────────┘ └───────────────────────┘
▲ │
│ On crash/rebalance │
└──────────────────────────────┘
Rebuild state from changelog
Key Takeaway
Stateful operations are what make Kafka Streams more than a simple message router. They let you accumulate results across multiple records, computing running counts, maximums, balances, and custom aggregations per key.
The workflow always starts with grouping. groupByKey() groups records by their existing key without repartitioning. groupBy() computes a new key and triggers repartitioning through an internal topic. The rule is to use groupByKey() whenever possible and reserve groupBy() for cases where you need to aggregate by a different key, like counting transactions per branch instead of per account.
After grouping, you choose an aggregation. count() is the simplest: it counts records per key. reduce() combines two values of the same type, like finding the largest transaction per account. aggregate() is the most flexible: it takes an initializer and an adder function, and can produce a result of any type. In our National Bank of India analytics project, we used all three: count for transaction counts, reduce for maximum transactions, and aggregate for running balances.
Every stateful operation creates a local state store (RocksDB by default) and a changelog topic in Kafka. The state store holds the current aggregated values. The changelog topic is the backup for recovery. If the application crashes or a partition rebalances to another instance, the state store is rebuilt from the changelog. At the senior level, the key concerns are naming state stores explicitly, monitoring their size, enabling standby replicas for faster recovery, and tuning the cache buffer to reduce changelog writes.
Remember one sentence: stateful operations group records by key and accumulate results in a local state store backed by a changelog topic, and the choice between groupByKey and groupBy determines whether repartitioning happens.
---