Back to Kafka Streams
Q16Question 16 of 27

What are state stores in Kafka Streams

Answer: A state store in Kafka Streams is local storage (RocksDB on disk or HashMap in memory) that holds accumulated state like account balances, backed by a changelog topic in Kafka for fault-tolerant recovery.

Key Concepts
  • A state store is local storage that a Kafka Streams application uses to hold aggregated or lookup data across multiple records.
  • Persistent stores use RocksDB on disk and survive application restarts; in-memory stores use a HashMap and are lost on restart.
  • Every state store is backed by a changelog topic in Kafka by default, which enables fault-tolerant recovery after a crash or rebalance.
  • RocksDB stores are the default choice for production because they handle datasets larger than available memory by spilling to disk.
  • In-memory stores are faster for small datasets (exchange rates, branch configs) but require full rebuild from the changelog on restart.
  • Interactive queries let you expose a state store as a read-only key-value lookup, turning your Kafka Streams app into a lightweight data service.
  • Standby replicas keep a warm copy of the state store on another instance to reduce recovery time after a rebalance.

Definition

A state store in Kafka Streams is local storage (RocksDB on disk or HashMap in memory) that holds accumulated state like account balances, backed by a changelog topic in Kafka for fault-tolerant recovery.
  • State store is local key-value storage inside a Kafka Streams application that holds aggregated or lookup data.
  • Persistent store (RocksDB) is the default state store type that writes data to disk using embedded RocksDB, surviving application restarts.
  • In-memory store is a state store backed by a Java HashMap that is faster but loses all data when the application stops.
  • Changelog topic is an internal, compacted Kafka topic that mirrors every write to the state store, enabling recovery.
  • Interactive queries is a Kafka Streams API that lets external clients read from a state store without consuming the output topic.
  • Standby replica is a warm copy of a state store maintained on a different application instance to reduce failover time.

Think of National Bank of India's branch ledger system. The main ledger book on the manager's desk is the persistent state store (RocksDB): it is written on paper, survives the office closing at night, and can hold millions of entries. A sticky note on the teller's monitor is the in-memory store: fast to read but lost when the monitor is turned off. The carbon copy sent to head office every day is the changelog topic: if the branch ledger is destroyed, head office sends back the carbon copies to rebuild it. The glass window where customers can see their balance without entering the back office is the interactive query: read-only access to the ledger from outside.

Understanding the Concepts

What are state stores in Kafka Streams?

A state store is local key-value storage that a Kafka Streams application uses to hold data across multiple records. When you call groupByKey().aggregate(), the accumulated result for each key is stored in a state store. When you create a KTable from a topic, the latest value per key is stored in a state store.

State stores are partitioned: each stream task owns a portion of the store, corresponding to the Kafka partitions it processes. Task 0 stores data for partition 0, task 1 for partition 1, and so on.

JAVA
package com.bank.streams;

public class StateStoreOverview {
    public static void main(String[] args) {
        System.out.println("State stores in Kafka Streams:");
        System.out.println("  Type 1: Persistent (RocksDB on disk)");
        System.out.println("  Type 2: In-memory (HashMap in JVM heap)");
        System.out.println();
        System.out.println("Backing:");
        System.out.println("  Every store has a changelog topic by default");
        System.out.println("  Changelog is compacted (only latest per key)");
        System.out.println();
        System.out.println("Partitioning:");
        System.out.println("  Each task owns its own partition of the store");
        System.out.println("  3 partitions = 3 state store instances");
    }
}
TEXT
Expected output:
State stores in Kafka Streams:
  Type 1: Persistent (RocksDB on disk)
  Type 2: In-memory (HashMap in JVM heap)

Backing:
  Every store has a changelog topic by default
  Changelog is compacted (only latest per key)

Partitioning:
  Each task owns its own partition of the store
  3 partitions = 3 state store instances

How does the persistent RocksDB store work?

The persistent store uses embedded RocksDB, a high-performance key-value database that writes to the local filesystem. Data is stored in SST (Sorted String Table) files on disk. RocksDB can hold datasets much larger than available memory because it reads from disk when needed.

When Kafka Streams starts, RocksDB opens the existing data files from the state directory (state.dir, default /tmp/kafka-streams). If the files are intact, no recovery from the changelog is needed. If the files are missing (new instance or deleted state), the store is rebuilt from the changelog topic.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.state.Stores;

public class PersistentStoreExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Aggregate with persistent RocksDB store (default)
        KTable<String, String> balances = transactions
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            .aggregate(
                () -> "0",
                (key, value, aggregate) -> {
                    String[] parts = value.split("\\|");
                    long amount = Long.parseLong(parts[2]);
                    long balance = Long.parseLong(aggregate);
                    if ("DEPOSIT".equals(parts[1])) {
                        balance += amount;
                    } else {
                        balance -= amount;
                    }
                    return String.valueOf(balance);
                },
                // Persistent RocksDB store (this is the default)
                Materialized.as("balance-store")
            );

        balances.toStream().to("account-balances");

        Topology topology = builder.build();
        System.out.println("Persistent store (RocksDB):");
        System.out.println("  Store name: balance-store");
        System.out.println("  Storage: disk (SST files)");
        System.out.println("  Location: /tmp/kafka-streams/<app-id>/<task-id>/");
        System.out.println("  Survives restart: YES (if state.dir is intact)");
    }
}
TEXT
Expected output:
Persistent store (RocksDB):
  Store name: balance-store
  Storage: disk (SST files)
  Location: /tmp/kafka-streams/<app-id>/<task-id>/
  Survives restart: YES (if state.dir is intact)

How does the in-memory store work?

The in-memory store uses a Java TreeMap (for sorted access) or HashMap stored entirely in JVM heap memory. It is faster than RocksDB because there is no disk I/O, but all data is lost when the application stops. On restart, the entire store must be rebuilt from the changelog topic.

Use in-memory stores for small reference datasets like exchange rates, branch configurations, or currency codes where the rebuild time is acceptable.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.state.Stores;
import org.apache.kafka.common.serialization.Serdes;

public class InMemoryStoreExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        // Read exchange rates into an in-memory KTable
        KTable<String, String> exchangeRates = builder.table(
            "exchange-rates",
            Materialized.<String, String>as(
                Stores.inMemoryKeyValueStore("rates-store")
            ).withKeySerde(Serdes.String())
             .withValueSerde(Serdes.String())
        );

        exchangeRates.toStream().foreach((pair, rate) ->
            System.out.println("Rate: " + pair + " = " + rate));

        Topology topology = builder.build();
        System.out.println("In-memory store:");
        System.out.println("  Store name: rates-store");
        System.out.println("  Storage: JVM heap (TreeMap)");
        System.out.println("  Survives restart: NO (rebuilt from changelog)");
        System.out.println("  Best for: small datasets < 100K entries");
    }
}
TEXT
Expected output:
Rate: INR_USD = 0.012
Rate: INR_EUR = 0.011
Rate: INR_GBP = 0.0095
In-memory store:
  Store name: rates-store
  Storage: JVM heap (TreeMap)
  Survives restart: NO (rebuilt from changelog)
  Best for: small datasets < 100K entries

How do persistent and in-memory stores compare?

FeaturePersistent (RocksDB)In-Memory
Storage mediumDisk (SST files)JVM heap (TreeMap)
Max dataset sizeLarger than memoryLimited by heap size
Read latencyMicroseconds (cached) to milliseconds (disk)Nanoseconds (always in memory)
Write latencyMicroseconds (write to WAL + memtable)Nanoseconds
Survives restartYes, if state.dir is intactNo, rebuilt from changelog
Recovery on clean restartInstant (opens existing files)Full rebuild from changelog
Recovery after rebalanceRestore from changelog (or standby)Full rebuild from changelog
Memory usageBlock cache (configurable, default 50MB)Entire dataset in heap
Banking use caseAccount balances (50M accounts)Exchange rates (200 pairs)
Default in Kafka StreamsYesNo (must specify explicitly)
JAVA
package com.bank.streams;

public class PersistentVsInMemoryComparison {
    public static void main(String[] args) {
        System.out.println("=== Persistent (RocksDB) ===");
        System.out.println("  50M accounts, 500 bytes each = ~25 GB on disk");
        System.out.println("  Block cache in memory: 50 MB (configurable)");
        System.out.println("  Restart: opens existing SST files instantly");
        System.out.println();
        System.out.println("=== In-Memory ===");
        System.out.println("  200 exchange rate pairs, 50 bytes each = ~10 KB");
        System.out.println("  All in JVM heap");
        System.out.println("  Restart: rebuild 200 entries from changelog in < 1 second");
        System.out.println();
        System.out.println("Rule: Use RocksDB (default) for large datasets.");
        System.out.println("Use in-memory only for small, fast-changing reference data.");
    }
}
TEXT
Expected output:
=== Persistent (RocksDB) ===
  50M accounts, 500 bytes each = ~25 GB on disk
  Block cache in memory: 50 MB (configurable)
  Restart: opens existing SST files instantly

=== In-Memory ===
  200 exchange rate pairs, 50 bytes each = ~10 KB
  All in JVM heap
  Restart: rebuild 200 entries from changelog in < 1 second

Rule: Use RocksDB (default) for large datasets.
Use in-memory only for small, fast-changing reference data.

What are changelog topics and how does state recovery work?

Every state store write is also written to a changelog topic in Kafka. The changelog topic is named {application.id}-{store-name}-changelog. It uses cleanup.policy=compact so only the latest value per key is retained, keeping the topic size bounded.

When a Kafka Streams instance loses its state (crash, rebalance, or deleted state directory), it replays the changelog topic from the beginning to rebuild the state store. The recovery time depends on the number of unique keys in the changelog.

JAVA
package com.bank.streams;

public class ChangelogRecoveryExample {
    public static void main(String[] args) {
        System.out.println("Changelog topic naming:");
        System.out.println("  Application ID: txn-analytics-service");
        System.out.println("  Store name:     balance-store");
        System.out.println("  Changelog:      txn-analytics-service-balance-store-changelog");
        System.out.println();
        System.out.println("Recovery scenario:");
        System.out.println("  1. App crashes, state directory deleted");
        System.out.println("  2. App restarts, state store is empty");
        System.out.println("  3. Kafka Streams reads changelog from beginning");
        System.out.println("  4. Each record restores one key-value pair");
        System.out.println("  5. Store fully rebuilt, processing resumes");
        System.out.println();
        System.out.println("Recovery time for 10M accounts:");
        System.out.println("  Changelog size: ~10M records (compacted)");
        System.out.println("  Estimated time: 2-5 minutes (depends on network)");
    }
}
TEXT
Expected output:
Changelog topic naming:
  Application ID: txn-analytics-service
  Store name:     balance-store
  Changelog:      txn-analytics-service-balance-store-changelog

Recovery scenario:
  1. App crashes, state directory deleted
  2. App restarts, state store is empty
  3. Kafka Streams reads changelog from beginning
  4. Each record restores one key-value pair
  5. Store fully rebuilt, processing resumes

Recovery time for 10M accounts:
  Changelog size: ~10M records (compacted)
  Estimated time: 2-5 minutes (depends on network)

What are interactive queries?

Interactive queries let you read from a state store without consuming the output topic. You call KafkaStreams.store() with the store name and type, and get a read-only view. This turns your Kafka Streams application into a lightweight key-value data service.

At National Bank of India, Rajesh exposes the account balance state store through a REST API. The branch manager queries the current balance for SAV2001 without consuming the account-balances topic.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StoreQueryParameters;
import org.apache.kafka.streams.state.QueryableStoreTypes;
import org.apache.kafka.streams.state.ReadOnlyKeyValueStore;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;

public class InteractiveQueryExample {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG,
                  "balance-query-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        // Assume BalanceLookupTopology.build() creates a KTable
        // with Materialized.as("balance-store")
        KafkaStreams streams = new KafkaStreams(
            BalanceLookupTopology.build(), props);
        streams.start();

        // Wait for the app to be in RUNNING state
        Thread.sleep(5000);

        // Query the state store directly
        ReadOnlyKeyValueStore<String, String> balanceStore =
            streams.store(
                StoreQueryParameters.fromNameAndType(
                    "balance-store",
                    QueryableStoreTypes.keyValueStore()
                )
            );

        // Lookup balance for SAV2001
        String balance = balanceStore.get("SAV2001");
        System.out.println("Balance for SAV2001: " + balance + " INR");

        // Lookup balance for CUR3001
        String balance2 = balanceStore.get("CUR3001");
        System.out.println("Balance for CUR3001: " + balance2 + " INR");

        // Key not found
        String balance3 = balanceStore.get("SAV9999");
        System.out.println("Balance for SAV9999: " + balance3);

        streams.close();
    }
}
TEXT
Expected output:
Balance for SAV2001: 290000 INR
Balance for CUR3001: 500000 INR
Balance for SAV9999: null

What are standby replicas?

Standby replicas are warm copies of state stores maintained on other application instances. When you set num.standby.replicas=1, Kafka Streams assigns one replica task to another instance that continuously reads the changelog topic and keeps its local state store up to date.

When a rebalance happens and a partition moves to a new instance, the new owner can use the standby replica instead of rebuilding from scratch. This reduces recovery time from minutes to seconds.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsConfig;

import java.util.Properties;

public class StandbyReplicaConfig {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG,
                  "balance-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        // Enable 1 standby replica per state store partition
        props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);

        System.out.println("Standby replica configuration:");
        System.out.println("  num.standby.replicas = 1");
        System.out.println();
        System.out.println("How it works:");
        System.out.println("  Instance A: owns partition 0, active store");
        System.out.println("  Instance B: standby for partition 0, reads changelog");
        System.out.println();
        System.out.println("When Instance A fails:");
        System.out.println("  Instance B already has the state store up to date");
        System.out.println("  Recovery time: seconds instead of minutes");
        System.out.println();
        System.out.println("Cost: double the disk usage and changelog read traffic");
    }
}
TEXT
Expected output:
Standby replica configuration:
  num.standby.replicas = 1

How it works:
  Instance A: owns partition 0, active store
  Instance B: standby for partition 0, reads changelog

When Instance A fails:
  Instance B already has the state store up to date
  Recovery time: seconds instead of minutes

Cost: double the disk usage and changelog read traffic

What is a complete banking project using state stores?

This project is an Account Balance Lookup Service for National Bank of India. It reads transactions, aggregates balances into a named state store, and exposes the store for interactive queries.

Project structure

DIAGRAM
balance-lookup-service/
├── pom.xml
└── src/
    └── main/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        ├── BalanceLookupApp.java
                        ├── BalanceLookupTopology.java
                        ├── BankTransaction.java
                        └── BalanceLookupTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>balance-lookup-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

BankTransaction.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class BankTransaction {
    private String accountId;
    private String type;
    private BigDecimal amount;
    private String description;

    public BankTransaction() {}

    public BankTransaction(String accountId, String type,
                           BigDecimal amount, String description) {
        this.accountId = accountId;
        this.type = type;
        this.amount = amount;
        this.description = description;
    }

    public String getAccountId() { return accountId; }
    public String getType() { return type; }
    public BigDecimal getAmount() { return amount; }
    public String getDescription() { return description; }

    @Override
    public String toString() {
        return accountId + "|" + type + "|" + amount + "|" + description;
    }

    public static BankTransaction fromString(String value) {
        String[] parts = value.split("\\|");
        return new BankTransaction(
            parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
    }
}

BalanceLookupTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.math.BigDecimal;

public class BalanceLookupTopology {

    public static final String BALANCE_STORE = "balance-store";

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Aggregate running balance into a named persistent state store
        KTable<String, String> balances = transactions
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            .aggregate(
                () -> "0",
                (accountId, txnValue, currentBalance) -> {
                    BankTransaction txn = BankTransaction.fromString(txnValue);
                    BigDecimal balance = new BigDecimal(currentBalance);
                    if ("DEPOSIT".equals(txn.getType())) {
                        balance = balance.add(txn.getAmount());
                    } else if ("WITHDRAW".equals(txn.getType())) {
                        balance = balance.subtract(txn.getAmount());
                    }
                    return balance.toPlainString();
                },
                Materialized.as(BALANCE_STORE)
            );

        // Also write to output topic for downstream consumers
        balances.toStream().to("account-balances");

        return builder.build();
    }
}

BalanceLookupApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.StoreQueryParameters;
import org.apache.kafka.streams.state.QueryableStoreTypes;
import org.apache.kafka.streams.state.ReadOnlyKeyValueStore;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.Scanner;
import java.util.concurrent.CountDownLatch;

public class BalanceLookupApp {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG,
                  "balance-lookup-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);

        KafkaStreams streams = new KafkaStreams(
            BalanceLookupTopology.build(), props);

        streams.start();
        System.out.println("Balance lookup service started");
        System.out.println("  Store: " + BalanceLookupTopology.BALANCE_STORE);
        System.out.println("  Standby replicas: 1");
        System.out.println("  Interactive queries: enabled");

        // Wait for RUNNING state
        Thread.sleep(5000);

        // Interactive query: lookup balance by account ID
        ReadOnlyKeyValueStore<String, String> store = streams.store(
            StoreQueryParameters.fromNameAndType(
                BalanceLookupTopology.BALANCE_STORE,
                QueryableStoreTypes.keyValueStore()
            )
        );

        System.out.println("\nLookup SAV2001: " + store.get("SAV2001") + " INR");
        System.out.println("Lookup CUR3001: " + store.get("CUR3001") + " INR");

        streams.close();
    }
}
TEXT
Expected output:
Balance lookup service started
  Store: balance-store
  Standby replicas: 1
  Interactive queries: enabled

Lookup SAV2001: 290000 INR
Lookup CUR3001: 500000 INR

BalanceLookupTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class BalanceLookupTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(BalanceLookupTopology.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "bank-transactions", new StringSerializer(), new StringSerializer());

            input.pipeInput("SAV2001", "SAV2001|DEPOSIT|250000|salary credit");
            input.pipeInput("SAV2001", "SAV2001|WITHDRAW|10000|atm withdrawal");
            input.pipeInput("CUR3001", "CUR3001|DEPOSIT|500000|vendor payment");

            // Query the state store directly via TopologyTestDriver
            KeyValueStore<String, String> store =
                driver.getKeyValueStore(BalanceLookupTopology.BALANCE_STORE);

            System.out.println("=== State Store Contents ===");
            System.out.println("SAV2001: " + store.get("SAV2001") + " INR");
            System.out.println("CUR3001: " + store.get("CUR3001") + " INR");
            System.out.println("SAV9999: " + store.get("SAV9999"));
            System.out.println();
            System.out.println("Store size: " + store.approximateNumEntries() + " entries");
        }
    }
}
TEXT
Expected output:
=== State Store Contents ===
SAV2001: 240000 INR
CUR3001: 500000 INR
SAV9999: null

Store size: 2 entries

How to run the project

BASH
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper

# Step 2: Create topics
kafka-topics.sh --create --topic bank-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic account-balances --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build and run
cd balance-lookup-service
mvn clean package
java -cp target/balance-lookup-service-1.0.0.jar com.bank.streams.BalanceLookupApp

# Step 4: Produce transactions
kafka-console-producer.sh --topic bank-transactions --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|DEPOSIT|250000|salary credit
# SAV2001:SAV2001|WITHDRAW|10000|atm withdrawal
# CUR3001:CUR3001|DEPOSIT|500000|vendor payment

What are common mistakes with state stores?

The most common mistake is not configuring the state directory for production.

Wrong: using default state.dir in production

JAVA
package com.bank.streams;

public class WrongDefaultStateDir {
    public static void main(String[] args) {
        System.out.println("WRONG: Using default state.dir in production");
        System.out.println();
        System.out.println("Default: state.dir = /tmp/kafka-streams");
        System.out.println();
        System.out.println("Problems:");
        System.out.println("  1. /tmp is cleared on OS reboot");
        System.out.println("  2. State store files are lost");
        System.out.println("  3. Full changelog replay needed on every reboot");
        System.out.println("  4. For 50M accounts, replay takes 5-10 minutes");
    }
}
TEXT
Expected output:
WRONG: Using default state.dir in production

Default: state.dir = /tmp/kafka-streams

Problems:
  1. /tmp is cleared on OS reboot
  2. State store files are lost
  3. Full changelog replay needed on every reboot
  4. For 50M accounts, replay takes 5-10 minutes

Correct: configuring state.dir on a persistent volume

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsConfig;

import java.util.Properties;

public class CorrectStateDir {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "balance-service");

        // Use a persistent volume for state storage
        props.put(StreamsConfig.STATE_DIR_CONFIG, "/data/kafka-streams");

        System.out.println("CORRECT: state.dir = /data/kafka-streams");
        System.out.println();
        System.out.println("Benefits:");
        System.out.println("  1. Survives OS reboot");
        System.out.println("  2. RocksDB opens existing files on restart");
        System.out.println("  3. No changelog replay needed");
        System.out.println("  4. Instant recovery");
        System.out.println();
        System.out.println("In Kubernetes: mount a PersistentVolumeClaim at /data/kafka-streams");
    }
}
TEXT
Expected output:
CORRECT: state.dir = /data/kafka-streams

Benefits:
  1. Survives OS reboot
  2. RocksDB opens existing files on restart
  3. No changelog replay needed
  4. Instant recovery

In Kubernetes: mount a PersistentVolumeClaim at /data/kafka-streams

What should a senior or principal engineer know?

At the senior level, you should understand RocksDB tuning, changelog management, and operational concerns for state stores in production.

  • RocksDB block cache: The default block cache is 50 MB per state store. For large stores, increase it to 256 MB or more. Set via StreamsConfig with a custom RocksDBConfigSetter.
  • Write buffer size: RocksDB writes go to an in-memory write buffer (memtable) before flushing to disk. Increasing write_buffer_size reduces write amplification but uses more memory.
  • Changelog retention: Changelog topics are compacted by default. If you also add a time-based retention (e.g., 7 days), records older than 7 days are deleted even if they are the latest for a key. This can cause data loss during recovery. Keep changelog topics as compact-only.
  • State store sizing formula: Total disk usage per store = (number of unique keys) x (average key size + average value size) x 1.5 (RocksDB overhead). For 50M accounts with 500-byte values, expect about 37.5 GB.
  • Monitoring: Expose JMX metrics for state store size (rocksdb.estimate-num-keys), block cache hit rate (rocksdb.block-cache-hit-ratio), and changelog consumer lag.
  • Disable changelog for read-only stores: If you use a GlobalKTable for reference data that is already replicated in Kafka, you can disable the changelog with Materialized.withLoggingDisabled() to save storage.
JAVA
package com.bank.streams;

public class SeniorStateStoreKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior checklist for state stores:");
        System.out.println("  1. Set state.dir to persistent volume, not /tmp");
        System.out.println("  2. Tune RocksDB block cache (default 50 MB per store)");
        System.out.println("  3. Keep changelog as compact-only, no time retention");
        System.out.println("  4. Enable standby replicas for fast failover");
        System.out.println("  5. Monitor: estimate-num-keys, block-cache-hit-ratio, changelog-lag");
        System.out.println("  6. Size formula: keys x (key + value size) x 1.5 overhead");
        System.out.println("  7. Disable changelog for GlobalKTable reference stores");
    }
}
TEXT
Expected output:
Senior checklist for state stores:
  1. Set state.dir to persistent volume, not /tmp
  2. Tune RocksDB block cache (default 50 MB per store)
  3. Keep changelog as compact-only, no time retention
  4. Enable standby replicas for fast failover
  5. Monitor: estimate-num-keys, block-cache-hit-ratio, changelog-lag
  6. Size formula: keys x (key + value size) x 1.5 overhead
  7. Disable changelog for GlobalKTable reference stores

What are common interview follow-up questions?

  • Can you use a state store without an aggregation? (Yes. You can add a custom state store to a topology using builder.addStateStore() and access it in a Processor or Transformer. This is useful for custom lookup tables or caches.)
  • What happens if the state store runs out of disk space? (RocksDB throws an IOException and the stream thread dies. The task enters ERROR state. You must free disk space or increase the volume before restarting.)
  • Can two applications share the same state store? (No. Each application.id has its own state stores and changelog topics. State stores are local to the application instance and partition.)
  • How does Kafka Streams decide where to restore state after a rebalance? (It uses the partition assignment strategy. If a partition moves from instance A to instance B, instance B checks if it has a standby replica. If yes, it promotes the standby. If no, it replays the changelog.)
  • Can you query a state store from a different machine? (Not directly. Interactive queries are local to the instance that owns the partition. For cross-instance queries, you need a discovery mechanism. Kafka Streams provides streams.metadataForKey() to find which instance hosts a specific key.)
  • What is the difference between Materialized.as("name") and Materialized.as(Stores.persistentKeyValueStore("name"))? (The first uses the default store type (persistent RocksDB). The second explicitly specifies persistent. They are equivalent. Use Stores.inMemoryKeyValueStore() if you want in-memory.)

Visual Overview

DIAGRAM
 State Store Architecture
 ==========================

 ┌─────────────────────────────────────────┐
 │ Kafka Streams Application               │
 │                                         │
 │  ┌─────────────┐   ┌─────────────┐      │
 │  │ Task 0      │   │ Task 1      │      │
 │  │ Partition 0  │   │ Partition 1  │      │
 │  │             │   │             │      │
 │  │ ┌─────────┐ │   │ ┌─────────┐ │      │
 │  │ │ State   │ │   │ │ State   │ │      │
 │  │ │ Store 0 │ │   │ │ Store 1 │ │      │
 │  │ │(RocksDB)│ │   │ │(RocksDB)│ │      │
 │  │ └────┬────┘ │   │ └────┬────┘ │      │
 │  └──────┼──────┘   └──────┼──────┘      │
 │         │                 │             │
 └─────────┼─────────────────┼─────────────┘
           │                 │
           ▼                 ▼
 ┌─────────────────────────────────────────┐
 │ Changelog Topic (Kafka)                 │
 │ app-id-store-name-changelog             │
 │ cleanup.policy = compact                │
 │                                         │
 │ Partition 0: SAV2001=290000             │
 │ Partition 1: CUR3001=500000             │
 └─────────────────────────────────────────┘


 Recovery Flow
 ===============

 Normal Restart (state.dir intact):
 ┌──────────┐    ┌──────────────┐
 │ App      │───→│ Open existing│───→ Ready (instant)
 │ starts   │    │ RocksDB files│
 └──────────┘    └──────────────┘

 State Lost (state.dir deleted):
 ┌──────────┐    ┌──────────────┐    ┌────────────┐
 │ App      │───→│ Empty store  │───→│ Replay     │───→ Ready
 │ starts   │    │ detected     │    │ changelog  │    (minutes)
 └──────────┘    └──────────────┘    └────────────┘

 With Standby Replica:
 ┌──────────┐    ┌──────────────┐
 │ Rebalance│───→│ Promote      │───→ Ready (seconds)
 │ detected │    │ standby      │
 └──────────┘    └──────────────┘

Key Takeaway

State stores are the foundation of stateful processing in Kafka Streams. Every aggregation, every KTable, and every join uses a state store to hold accumulated data. Understanding how they work is essential for building reliable stream processing applications.

The persistent RocksDB store is the default and the right choice for most production workloads. It writes data to disk as SST files, handles datasets much larger than available memory, and survives application restarts when the state directory is on a persistent volume. The in-memory store is faster but loses all data on shutdown, making it suitable only for small reference datasets like exchange rates or branch configurations that can be rebuilt quickly from the changelog.

Every state store is backed by a changelog topic in Kafka. This topic is compacted, keeping only the latest value per key. When a state store is lost (crash, rebalance, or deleted state directory), it is rebuilt by replaying the changelog from the beginning. Standby replicas keep a warm copy on another instance, reducing recovery time from minutes to seconds. At National Bank of India, Rajesh configures one standby replica for the balance store so that branch managers never wait more than a few seconds during failover.

Interactive queries turn a Kafka Streams application into a lightweight data service. By calling KafkaStreams.store() with the store name, you get a read-only view of the state store. The branch manager can query the current balance for SAV2001 without consuming the output topic.

At the senior level, the critical concerns are: set state.dir to a persistent volume (not /tmp), tune RocksDB block cache for your dataset size, keep changelog topics as compact-only without time-based retention, and monitor state store size and changelog consumer lag via JMX.

Remember one sentence: a state store is local key-value storage backed by a changelog topic for recovery, and the choice between persistent RocksDB and in-memory determines whether your data survives a restart or must be rebuilt.

---

Share
Use ← → arrow keys to navigate