Back to Kafka Streams
Q15Question 15 of 27

What are stateful operations in Kafka Streams

Answer: Stateful operations in Kafka Streams maintain local state stores to accumulate results across multiple records, and include groupByKey, groupBy, count, reduce, and aggregate for computing running totals, counts, and custom aggregations per key.

Key Concepts
  • Stateful operations maintain local state stores to accumulate results across multiple records, unlike stateless operations that process each record independently.
  • groupByKey() groups records by their existing key without repartitioning; groupBy() groups records by a new key and triggers repartitioning.
  • count() counts the number of records per key and stores the running count in a state store backed by a changelog topic.
  • reduce() combines records with the same key using a binary function, producing a single accumulated value per key.
  • aggregate() is the most flexible stateful operation: it takes an initializer, an adder function, and a Materialized store to produce any type of accumulated result per key.
  • Every stateful operation creates a local state store (RocksDB by default) and a changelog topic in Kafka for fault-tolerant recovery.
  • Windowed versions of count, reduce, and aggregate operate over time-bounded groups using tumbling, hopping, or session windows.

Definition

Stateful operations in Kafka Streams maintain local state stores to accumulate results across multiple records, and include groupByKey, groupBy, count, reduce, and aggregate for computing running totals, counts, and custom aggregations per key.
  • Stateful operation is a stream processing step that maintains a local state store to accumulate information across multiple records with the same key.
  • groupByKey() groups records by their current key without changing it, which avoids repartitioning.
  • groupBy() groups records by a new key computed from the record, which triggers repartitioning because the key changes.
  • count() is an aggregation that counts the number of records per grouped key and stores the count in a state store.
  • reduce() is an aggregation that combines two values into one using a binary function, producing one accumulated result per key.
  • aggregate() is the most general aggregation: it takes an initializer for new keys, an adder function, and a Materialized store configuration.
  • Changelog topic is an internal Kafka topic that backs each state store for fault-tolerant recovery after a crash or rebalance.

Think of the daily reporting desk at National Bank of India. Each clerk at the desk keeps a register (state store) to track running totals. One clerk counts the number of transactions per account (count). Another clerk tracks the largest single transaction per account (reduce). A third clerk maintains a running balance per account, adding deposits and subtracting withdrawals (aggregate). If a clerk's register is destroyed, they rebuild it from the carbon copies stored in the filing cabinet (changelog topic). The groupByKey step is like sorting slips by account number before handing them to the clerks.

Understanding the Concepts

What are stateful operations in Kafka Streams?

Stateful operations process records while maintaining local state that persists across multiple records. Unlike stateless operations that forget everything after processing a record, stateful operations remember accumulated results in a state store.

Every stateful operation creates two things: a local state store (RocksDB by default) and a changelog topic in Kafka. The state store holds the current aggregated values. The changelog topic is a backup: if the application crashes or a partition rebalances to another instance, the state store is rebuilt from the changelog.

JAVA
package com.bank.streams;

public class StatefulOperationsOverview {
    public static void main(String[] args) {
        System.out.println("Stateful operations in Kafka Streams:");
        System.out.println("  groupByKey -> count   : transactions per account");
        System.out.println("  groupByKey -> reduce  : max transaction per account");
        System.out.println("  groupByKey -> aggregate: running balance per account");
        System.out.println("  groupBy    -> count   : transactions per branch");
        System.out.println();
        System.out.println("Each creates:");
        System.out.println("  1. Local state store (RocksDB)");
        System.out.println("  2. Changelog topic (for recovery)");
    }
}
TEXT
Expected output:
Stateful operations in Kafka Streams:
  groupByKey -> count   : transactions per account
  groupByKey -> reduce  : max transaction per account
  groupByKey -> aggregate: running balance per account
  groupBy    -> count   : transactions per branch

Each creates:
  1. Local state store (RocksDB)
  2. Changelog topic (for recovery)

How does groupByKey work?

groupByKey() groups records by their existing key. It returns a KGroupedStream, which you can then pass to count(), reduce(), or aggregate(). Because the key does not change, no repartitioning is needed.

At National Bank of India, transactions arrive keyed by account ID (SAV2001, CUR3001). Rajesh calls groupByKey() to group all transactions by account before counting or aggregating them.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KGroupedStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.Topology;

public class GroupByKeyExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Group by existing key (account ID). No repartition needed.
        KGroupedStream<String, String> grouped = transactions.groupByKey();

        // Count transactions per account
        KTable<String, Long> txnCounts = grouped.count();

        txnCounts.toStream().foreach((accountId, count) ->
            System.out.println("Account " + accountId + ": " + count + " transactions"));

        Topology topology = builder.build();
        System.out.println("groupByKey topology (no repartition):");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Account SAV2001: 1 transactions
Account SAV2001: 2 transactions
Account CUR3001: 1 transactions
Account SAV2001: 3 transactions
groupByKey topology (no repartition):
  No repartition topic in the topology description.

How does groupBy work?

groupBy() groups records by a new key computed from the record. Because the key changes, the resulting stream is repartitioned through an internal topic. Use groupBy() when you need to aggregate by a different key than the record's current key.

At National Bank of India, transactions arrive keyed by account ID. Rajesh needs to count transactions per branch. The branch code is the first three characters of the account ID (SAV, CUR). He uses groupBy() to rekey by branch code.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KGroupedStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

public class GroupByExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Group by branch code (new key). Triggers repartitioning.
        KGroupedStream<String, String> byBranch = transactions.groupBy(
            (accountId, value) -> accountId.substring(0, 3),
            Grouped.with(Serdes.String(), Serdes.String())
        );

        // Count transactions per branch
        KTable<String, Long> branchCounts = byBranch.count();

        branchCounts.toStream().foreach((branch, count) ->
            System.out.println("Branch " + branch + ": " + count + " transactions"));

        Topology topology = builder.build();
        System.out.println("groupBy topology (with repartition):");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Branch SAV: 1 transactions
Branch SAV: 2 transactions
Branch CUR: 1 transactions
Branch SAV: 3 transactions
groupBy topology (with repartition):
  Contains internal repartition topic in topology description.

How do groupByKey and groupBy compare?

The key difference is whether the grouping key changes. groupByKey uses the existing key; groupBy computes a new key.

FeaturegroupByKeygroupBy
Key changeNo, uses existing keyYes, computes new key
RepartitioningNoYes, creates internal repartition topic
PerformanceFaster, no network overheadSlower, records written to repartition topic
When to useAggregate by the same keyAggregate by a different key
Banking exampleCount per account (key=account)Count per branch (key=branch code)
Serde requirementUses default serdesMust specify serdes via Grouped.with()
JAVA
package com.bank.streams;

public class GroupByKeyVsGroupBy {
    public static void main(String[] args) {
        System.out.println("=== groupByKey ===");
        System.out.println("  Input key: SAV2001 (account ID)");
        System.out.println("  Group key: SAV2001 (same, no change)");
        System.out.println("  Repartition: NO");
        System.out.println("  Use case: count transactions per account");
        System.out.println();
        System.out.println("=== groupBy ===");
        System.out.println("  Input key: SAV2001 (account ID)");
        System.out.println("  Group key: SAV (branch code, new key)");
        System.out.println("  Repartition: YES");
        System.out.println("  Use case: count transactions per branch");
        System.out.println();
        System.out.println("Rule: Use groupByKey when possible. Use groupBy only");
        System.out.println("when you need to change the grouping key.");
    }
}
TEXT
Expected output:
=== groupByKey ===
  Input key: SAV2001 (account ID)
  Group key: SAV2001 (same, no change)
  Repartition: NO
  Use case: count transactions per account

=== groupBy ===
  Input key: SAV2001 (account ID)
  Group key: SAV (branch code, new key)
  Repartition: YES
  Use case: count transactions per branch

Rule: Use groupByKey when possible. Use groupBy only
when you need to change the grouping key.

How does count work?

count() counts the number of records per key in a grouped stream. It returns a KTable where the value is the running count. Each time a new record arrives for a key, the count increments and the updated value is emitted.

At National Bank of India, Rajesh counts the number of transactions per account per day to detect accounts with unusually high activity.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;

public class CountExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Count transactions per account
        KTable<String, Long> txnCounts = transactions
            .groupByKey()
            .count(Materialized.as("txn-count-store"));

        // Emit count changes
        txnCounts.toStream().foreach((accountId, count) ->
            System.out.println(accountId + ": " + count + " transactions"));

        Topology topology = builder.build();
        System.out.println("Count topology with named state store: txn-count-store");
    }
}
TEXT
Expected output (records arrive one by one):
SAV2001: 1 transactions
SAV2001: 2 transactions
CUR3001: 1 transactions
SAV2001: 3 transactions
Count topology with named state store: txn-count-store

How does reduce work?

reduce() combines records with the same key using a binary function. The function takes two values and returns one. It runs repeatedly: the first record becomes the initial value, and each subsequent record is combined with the accumulated result.

At National Bank of India, Rajesh uses reduce to find the largest transaction per account. The reduce function compares two transaction amounts and keeps the larger one.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class ReduceExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Reduce to find max transaction amount per account
        KTable<String, String> maxPerAccount = transactions
            .groupByKey()
            .reduce(
                (currentMax, newValue) -> {
                    BigDecimal current = new BigDecimal(currentMax.split("\\|")[2]);
                    BigDecimal incoming = new BigDecimal(newValue.split("\\|")[2]);
                    return incoming.compareTo(current) > 0 ? newValue : currentMax;
                },
                Materialized.as("max-txn-store")
            );

        maxPerAccount.toStream().foreach((accountId, maxTxn) ->
            System.out.println("Max for " + accountId + ": " + maxTxn));

        Topology topology = builder.build();
        System.out.println("Reduce topology: find max transaction per account");
    }
}
TEXT
Expected output:
Max for SAV2001: SAV2001|DEPOSIT|50000|salary credit
Max for SAV2001: SAV2001|NEFT|250000|property payment
Max for CUR3001: CUR3001|RTGS|500000|vendor payment
Max for SAV2001: SAV2001|NEFT|250000|property payment
Reduce topology: find max transaction per account

After 3 records for SAV2001 (50000, 250000, 10000):
  Max stays at 250000 because 10000 < 250000.

How does aggregate work?

aggregate() is the most flexible stateful operation. It takes three parameters: an initializer that creates the starting value for a new key, an adder function that combines the current aggregate with a new record, and a Materialized store configuration.

Unlike reduce(), aggregate() can produce a result of a different type than the input. The initializer runs once per new key. The adder runs for every subsequent record.

At National Bank of India, Rajesh uses aggregate to maintain a running balance per account. Deposits are added and withdrawals are subtracted.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.math.BigDecimal;

public class AggregateExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Aggregate running balance per account
        KTable<String, String> balances = transactions
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            .aggregate(
                // Initializer: starting balance for new accounts
                () -> "0",
                // Adder: update balance based on transaction type
                (accountId, txnValue, currentBalance) -> {
                    String[] parts = txnValue.split("\\|");
                    String type = parts[1];
                    BigDecimal amount = new BigDecimal(parts[2]);
                    BigDecimal balance = new BigDecimal(currentBalance);

                    if ("DEPOSIT".equals(type)) {
                        balance = balance.add(amount);
                    } else if ("WITHDRAW".equals(type)) {
                        balance = balance.subtract(amount);
                    }
                    return balance.toPlainString();
                },
                // Store configuration
                Materialized.as("balance-store")
            );

        balances.toStream().foreach((accountId, balance) ->
            System.out.println("Balance " + accountId + ": " + balance + " INR"));

        Topology topology = builder.build();
        System.out.println("Aggregate topology: running balance per account");
    }
}
TEXT
Expected output:
Balance SAV2001: 50000 INR     (DEPOSIT 50000)
Balance SAV2001: 40000 INR     (WITHDRAW 10000)
Balance SAV2001: 65000 INR     (DEPOSIT 25000)
Balance CUR3001: 500000 INR    (DEPOSIT 500000)
Aggregate topology: running balance per account

What are windowed aggregations?

Windowed aggregations group records by key and time window. Instead of aggregating all records for a key forever, you aggregate only records that fall within a specific time range.

  • Kafka Streams supports three window types:
  • Tumbling window: fixed-size, non-overlapping (e.g., every 1 hour)
  • Hopping window: fixed-size, overlapping (e.g., 1-hour window every 15 minutes)
  • Session window: variable-size, based on activity gaps (e.g., group events with gaps less than 30 minutes)

At National Bank of India, Rajesh counts transactions per account per hour to detect suspicious burst activity.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class WindowedCountExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Count transactions per account per 1-hour tumbling window
        KTable<Windowed<String>, Long> hourlyCounts = transactions
            .groupByKey()
            .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
            .count(Materialized.as("hourly-count-store"));

        hourlyCounts.toStream().foreach((windowedKey, count) -> {
            String accountId = windowedKey.key();
            long windowStart = windowedKey.window().start();
            long windowEnd = windowedKey.window().end();
            System.out.println(accountId + " [" + windowStart + "-" + windowEnd + "]: "
                + count + " transactions");
        });

        Topology topology = builder.build();
        System.out.println("Windowed count: transactions per account per hour");
    }
}
TEXT
Expected output:
SAV2001 [1700000000000-1700003600000]: 1 transactions
SAV2001 [1700000000000-1700003600000]: 2 transactions
SAV2001 [1700000000000-1700003600000]: 3 transactions
CUR3001 [1700000000000-1700003600000]: 1 transactions
Windowed count: transactions per account per hour

All records fell within the same 1-hour window.
If a record arrives in the next hour, a new window starts.

What is a complete banking project using stateful operations?

This project is a Daily Transaction Analytics Service for National Bank of India. It reads transactions and produces three outputs: transaction count per account, maximum transaction per account, and running balance per account.

Project structure

DIAGRAM
txn-analytics-service/
├── pom.xml
└── src/
    └── main/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        ├── TxnAnalyticsApp.java
                        ├── TxnAnalyticsTopology.java
                        ├── Transaction.java
                        └── TxnAnalyticsTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>txn-analytics-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

Transaction.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class Transaction {
    private String accountId;
    private String type;
    private BigDecimal amount;
    private String description;

    public Transaction() {}

    public Transaction(String accountId, String type,
                       BigDecimal amount, String description) {
        this.accountId = accountId;
        this.type = type;
        this.amount = amount;
        this.description = description;
    }

    public String getAccountId() { return accountId; }
    public String getType() { return type; }
    public BigDecimal getAmount() { return amount; }
    public String getDescription() { return description; }

    @Override
    public String toString() {
        return accountId + "|" + type + "|" + amount + "|" + description;
    }

    public static Transaction fromString(String value) {
        String[] parts = value.split("\\|");
        return new Transaction(
            parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
    }
}

TxnAnalyticsTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.math.BigDecimal;

public class TxnAnalyticsTopology {

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // 1. Count transactions per account
        KTable<String, Long> txnCounts = transactions
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            .count(Materialized.as("txn-count-store"));
        txnCounts.toStream()
            .mapValues(count -> String.valueOf(count))
            .to("txn-counts");

        // 2. Reduce to find max transaction per account
        KTable<String, String> maxTxn = transactions
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            .reduce(
                (current, incoming) -> {
                    BigDecimal currentAmt = new BigDecimal(current.split("\\|")[2]);
                    BigDecimal incomingAmt = new BigDecimal(incoming.split("\\|")[2]);
                    return incomingAmt.compareTo(currentAmt) > 0 ? incoming : current;
                },
                Materialized.as("max-txn-store")
            );
        maxTxn.toStream().to("max-transactions");

        // 3. Aggregate running balance per account
        KTable<String, String> balances = transactions
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            .aggregate(
                () -> "0",
                (accountId, txnValue, currentBalance) -> {
                    Transaction txn = Transaction.fromString(txnValue);
                    BigDecimal balance = new BigDecimal(currentBalance);
                    if ("DEPOSIT".equals(txn.getType())) {
                        balance = balance.add(txn.getAmount());
                    } else if ("WITHDRAW".equals(txn.getType())) {
                        balance = balance.subtract(txn.getAmount());
                    }
                    return balance.toPlainString();
                },
                Materialized.as("balance-store")
            );
        balances.toStream().to("account-balances");

        return builder.build();
    }
}

TxnAnalyticsApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class TxnAnalyticsApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG,
                  "txn-analytics-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        KafkaStreams streams = new KafkaStreams(
            TxnAnalyticsTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
        }));

        try {
            streams.start();
            System.out.println("Transaction analytics service started");
            System.out.println("  Input:   bank-transactions");
            System.out.println("  Output:  txn-counts, max-transactions, account-balances");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}
TEXT
Expected output (on startup):
Transaction analytics service started
  Input:   bank-transactions
  Output:  txn-counts, max-transactions, account-balances

TxnAnalyticsTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.LongDeserializer;

import java.util.Properties;

public class TxnAnalyticsTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(TxnAnalyticsTopology.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "bank-transactions", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> counts = driver.createOutputTopic(
                "txn-counts", new StringDeserializer(), new StringDeserializer());
            TestOutputTopic<String, String> maxTxns = driver.createOutputTopic(
                "max-transactions", new StringDeserializer(), new StringDeserializer());
            TestOutputTopic<String, String> balances = driver.createOutputTopic(
                "account-balances", new StringDeserializer(), new StringDeserializer());

            input.pipeInput("SAV2001", "SAV2001|DEPOSIT|50000|salary credit");
            input.pipeInput("SAV2001", "SAV2001|NEFT|250000|property payment");
            input.pipeInput("SAV2001", "SAV2001|WITHDRAW|10000|atm withdrawal");
            input.pipeInput("CUR3001", "CUR3001|DEPOSIT|500000|vendor payment");

            System.out.println("=== Transaction Counts ===");
            while (!counts.isEmpty()) {
                var r = counts.readKeyValue();
                System.out.println("  " + r.key + ": " + r.value);
            }

            System.out.println("=== Max Transactions ===");
            while (!maxTxns.isEmpty()) {
                var r = maxTxns.readKeyValue();
                System.out.println("  " + r.key + ": " + r.value);
            }

            System.out.println("=== Balances ===");
            while (!balances.isEmpty()) {
                var r = balances.readKeyValue();
                System.out.println("  " + r.key + ": " + r.value + " INR");
            }
        }
    }
}
TEXT
Expected output:
=== Transaction Counts ===
  SAV2001: 1
  SAV2001: 2
  SAV2001: 3
  CUR3001: 1
=== Max Transactions ===
  SAV2001: SAV2001|DEPOSIT|50000|salary credit
  SAV2001: SAV2001|NEFT|250000|property payment
  SAV2001: SAV2001|NEFT|250000|property payment
  CUR3001: CUR3001|DEPOSIT|500000|vendor payment
=== Balances ===
  SAV2001: 50000 INR
  SAV2001: 300000 INR
  SAV2001: 290000 INR
  CUR3001: 500000 INR

How to run the project

BASH
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper

# Step 2: Create topics
kafka-topics.sh --create --topic bank-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic txn-counts --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic max-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic account-balances --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build and run
cd txn-analytics-service
mvn clean package
java -cp target/txn-analytics-service-1.0.0.jar com.bank.streams.TxnAnalyticsApp

# Step 4: Produce test messages
kafka-console-producer.sh --topic bank-transactions --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|DEPOSIT|50000|salary credit
# SAV2001:SAV2001|NEFT|250000|property payment
# SAV2001:SAV2001|WITHDRAW|10000|atm withdrawal
# CUR3001:CUR3001|DEPOSIT|500000|vendor payment

What are common mistakes with stateful operations?

The most common mistake is using groupBy() when groupByKey() is sufficient, which triggers an unnecessary repartition.

Wrong: using groupBy with the same key

JAVA
package com.bank.streams;

public class WrongGroupByWithSameKey {
    public static void main(String[] args) {
        System.out.println("WRONG: Using groupBy() when key does not change");
        System.out.println();
        System.out.println("Code:");
        System.out.println("  transactions.groupBy(");
        System.out.println("      (key, value) -> key,");
        System.out.println("      Grouped.with(Serdes.String(), Serdes.String()))");
        System.out.println("  .count();");
        System.out.println();
        System.out.println("Problem: groupBy creates a repartition topic even though");
        System.out.println("the key is the same. This wastes network, storage, and time.");
    }
}
TEXT
Expected output:
WRONG: Using groupBy() when key does not change

Code:
  transactions.groupBy(
      (key, value) -> key,
      Grouped.with(Serdes.String(), Serdes.String()))
  .count();

Problem: groupBy creates a repartition topic even though
the key is the same. This wastes network, storage, and time.

Correct: using groupByKey when key stays the same

JAVA
package com.bank.streams;

public class CorrectGroupByKey {
    public static void main(String[] args) {
        System.out.println("CORRECT: Using groupByKey() when key does not change");
        System.out.println();
        System.out.println("Code:");
        System.out.println("  transactions.groupByKey()");
        System.out.println("  .count();");
        System.out.println();
        System.out.println("No repartition topic created.");
        System.out.println("Same result, better performance.");
    }
}
TEXT
Expected output:
CORRECT: Using groupByKey() when key does not change

Code:
  transactions.groupByKey()
  .count();

No repartition topic created.
Same result, better performance.

Wrong: not naming the state store

JAVA
package com.bank.streams;

public class WrongUnnamedStore {
    public static void main(String[] args) {
        System.out.println("WRONG: Not naming the state store");
        System.out.println();
        System.out.println("Code:");
        System.out.println("  transactions.groupByKey().count();");
        System.out.println();
        System.out.println("Problem: Kafka Streams assigns a generated name like");
        System.out.println("KSTREAM-AGGREGATE-STATE-STORE-0000000003.");
        System.out.println("If you add or remove operations, the generated name changes.");
        System.out.println("This makes the old changelog topic incompatible.");
        System.out.println("State cannot be restored from the old changelog.");
        System.out.println();
        System.out.println("CORRECT: Name the store explicitly");
        System.out.println("  transactions.groupByKey().count(Materialized.as(\"txn-count-store\"));");
        System.out.println("The changelog topic name is now stable across topology changes.");
    }
}
TEXT
Expected output:
WRONG: Not naming the state store

Code:
  transactions.groupByKey().count();

Problem: Kafka Streams assigns a generated name like
KSTREAM-AGGREGATE-STATE-STORE-0000000003.
If you add or remove operations, the generated name changes.
This makes the old changelog topic incompatible.
State cannot be restored from the old changelog.

CORRECT: Name the store explicitly
  transactions.groupByKey().count(Materialized.as("txn-count-store"));
The changelog topic name is now stable across topology changes.

What should a senior or principal engineer know?

At the senior level, you should understand how state stores and changelog topics affect your application's performance, storage, and recovery time.

  • Changelog compaction: Changelog topics use cleanup.policy=compact by default. This keeps only the latest value per key, limiting storage growth. Never change this to delete, or state recovery after a crash will be incomplete.
  • State store sizing: The state store holds one entry per unique key. If you aggregate 50 million accounts, the RocksDB store holds 50 million entries. Monitor the state store size via JMX metrics.
  • Recovery time: After a rebalance, the state store is rebuilt from the changelog topic. Recovery time is proportional to the number of entries. Use standby replicas (num.standby.replicas=1) to keep a warm copy on another instance.
  • Caching: Kafka Streams caches state store updates and flushes them periodically (cache.max.bytes.buffering). This reduces the number of records written to the changelog topic and emitted downstream. Set this to 10MB or more in production.
  • Exactly-once with aggregations: When processing.guarantee=exactly_once_v2, each aggregate update is part of a Kafka transaction. This ensures that the state store update and the changelog write are atomic.
  • Multiple aggregations: If you call groupByKey().count() and groupByKey().reduce() on the same stream, two separate state stores and two changelog topics are created. Each groupByKey() call creates an independent processing path.
JAVA
package com.bank.streams;

public class SeniorStatefulKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior checklist for stateful operations:");
        System.out.println("  1. Always name state stores with Materialized.as()");
        System.out.println("  2. Monitor state store size via JMX");
        System.out.println("  3. Set num.standby.replicas=1 for faster recovery");
        System.out.println("  4. Set cache.max.bytes.buffering >= 10MB in production");
        System.out.println("  5. Keep changelog compaction policy as compact");
        System.out.println("  6. Multiple aggregations = multiple stores + changelogs");
        System.out.println("  7. Use exactly_once_v2 for atomic state + changelog updates");
    }
}
TEXT
Expected output:
Senior checklist for stateful operations:
  1. Always name state stores with Materialized.as()
  2. Monitor state store size via JMX
  3. Set num.standby.replicas=1 for faster recovery
  4. Set cache.max.bytes.buffering >= 10MB in production
  5. Keep changelog compaction policy as compact
  6. Multiple aggregations = multiple stores + changelogs
  7. Use exactly_once_v2 for atomic state + changelog updates

What are common interview follow-up questions?

  • What is the difference between reduce and aggregate? (reduce combines two values of the same type; aggregate can produce a different type because it takes an initializer. Use reduce for simple combinations like max or sum; use aggregate when the result type differs from the input.)
  • Can you do a stateful operation without groupByKey or groupBy? (No. Stateful operations like count, reduce, and aggregate require a grouped stream. You must call groupByKey() or groupBy() first.)
  • What happens if a record has a null key? (Records with null keys are dropped by groupByKey() and groupBy(). They cannot be grouped because there is no key to group by. Always filter out null keys before grouping.)
  • How do windowed aggregations handle late-arriving records? (By default, late records outside the window's grace period are dropped. Use .windowedBy(TimeWindows.ofSizeAndGrace()) to set a grace period. Records arriving within the grace period update the window.)
  • Can you query the state store while the app is running? (Yes, using Interactive Queries. Call streams.store(StoreQueryParameters.fromNameAndType()) to get a read-only view of the store.)
  • What is the difference between count on KGroupedStream and KGroupedTable? (KGroupedStream.count() increments for every record. KGroupedTable.count() adjusts: it subtracts for the old value and adds for the new value when a key is updated.)

Visual Overview

DIAGRAM
 Stateful Operations Flow
 ==========================

 ┌───────────────────────────┐
 │ bank-transactions          │
 │  SAV2001|DEPOSIT|50000     │
 │  SAV2001|NEFT|250000       │
 │  SAV2001|WITHDRAW|10000    │
 │  CUR3001|DEPOSIT|500000    │
 └────────────┬──────────────┘
              │
              ▼
 ┌───────────────────────────┐
 │ groupByKey()              │
 │ (no repartition)          │
 └──────┬──────┬──────┬──────┘
        │      │      │
        ▼      ▼      ▼
 ┌────────┐┌────────┐┌──────────┐
 │ count()││reduce()││aggregate()│
 └───┬────┘└───┬────┘└─────┬────┘
     │         │           │
     ▼         ▼           ▼
 ┌────────┐┌────────┐┌──────────┐
 │ State  ││ State  ││ State    │
 │ Store  ││ Store  ││ Store    │
 │ (count)││ (max)  ││ (balance)│
 └───┬────┘└───┬────┘└─────┬────┘
     │         │           │
     ▼         ▼           ▼
 ┌────────┐┌────────┐┌──────────┐
 │txn-    ││max-    ││account-  │
 │counts  ││txns    ││balances  │
 └────────┘└────────┘└──────────┘


 State Store + Changelog
 =========================

 ┌──────────────────────┐
 │ Aggregation Operation │
 └──────────┬───────────┘
            │ write
            ▼
 ┌──────────────────────┐     ┌───────────────────────┐
 │ Local State Store    │────→│ Changelog Topic       │
 │ (RocksDB)            │     │ (compact, in Kafka)   │
 │ SAV2001: 290000      │     │ SAV2001: 290000       │
 │ CUR3001: 500000      │     │ CUR3001: 500000       │
 └──────────────────────┘     └───────────────────────┘
         ▲                              │
         │          On crash/rebalance  │
         └──────────────────────────────┘
              Rebuild state from changelog

Key Takeaway

Stateful operations are what make Kafka Streams more than a simple message router. They let you accumulate results across multiple records, computing running counts, maximums, balances, and custom aggregations per key.

The workflow always starts with grouping. groupByKey() groups records by their existing key without repartitioning. groupBy() computes a new key and triggers repartitioning through an internal topic. The rule is to use groupByKey() whenever possible and reserve groupBy() for cases where you need to aggregate by a different key, like counting transactions per branch instead of per account.

After grouping, you choose an aggregation. count() is the simplest: it counts records per key. reduce() combines two values of the same type, like finding the largest transaction per account. aggregate() is the most flexible: it takes an initializer and an adder function, and can produce a result of any type. In our National Bank of India analytics project, we used all three: count for transaction counts, reduce for maximum transactions, and aggregate for running balances.

Every stateful operation creates a local state store (RocksDB by default) and a changelog topic in Kafka. The state store holds the current aggregated values. The changelog topic is the backup for recovery. If the application crashes or a partition rebalances to another instance, the state store is rebuilt from the changelog. At the senior level, the key concerns are naming state stores explicitly, monitoring their size, enabling standby replicas for faster recovery, and tuning the cache buffer to reduce changelog writes.

Remember one sentence: stateful operations group records by key and accumulate results in a local state store backed by a changelog topic, and the choice between groupByKey and groupBy determines whether repartitioning happens.

---

Share
Use ← → arrow keys to navigate