Definition
- State store is local key-value storage inside a Kafka Streams application that holds aggregated or lookup data.
- Persistent store (RocksDB) is the default state store type that writes data to disk using embedded RocksDB, surviving application restarts.
- In-memory store is a state store backed by a Java HashMap that is faster but loses all data when the application stops.
- Changelog topic is an internal, compacted Kafka topic that mirrors every write to the state store, enabling recovery.
- Interactive queries is a Kafka Streams API that lets external clients read from a state store without consuming the output topic.
- Standby replica is a warm copy of a state store maintained on a different application instance to reduce failover time.
Think of National Bank of India's branch ledger system. The main ledger book on the manager's desk is the persistent state store (RocksDB): it is written on paper, survives the office closing at night, and can hold millions of entries. A sticky note on the teller's monitor is the in-memory store: fast to read but lost when the monitor is turned off. The carbon copy sent to head office every day is the changelog topic: if the branch ledger is destroyed, head office sends back the carbon copies to rebuild it. The glass window where customers can see their balance without entering the back office is the interactive query: read-only access to the ledger from outside.
Understanding the Concepts
What are state stores in Kafka Streams?
A state store is local key-value storage that a Kafka Streams application uses to hold data across multiple records. When you call groupByKey().aggregate(), the accumulated result for each key is stored in a state store. When you create a KTable from a topic, the latest value per key is stored in a state store.
State stores are partitioned: each stream task owns a portion of the store, corresponding to the Kafka partitions it processes. Task 0 stores data for partition 0, task 1 for partition 1, and so on.
package com.bank.streams;
public class StateStoreOverview {
public static void main(String[] args) {
System.out.println("State stores in Kafka Streams:");
System.out.println(" Type 1: Persistent (RocksDB on disk)");
System.out.println(" Type 2: In-memory (HashMap in JVM heap)");
System.out.println();
System.out.println("Backing:");
System.out.println(" Every store has a changelog topic by default");
System.out.println(" Changelog is compacted (only latest per key)");
System.out.println();
System.out.println("Partitioning:");
System.out.println(" Each task owns its own partition of the store");
System.out.println(" 3 partitions = 3 state store instances");
}
}
Expected output:
State stores in Kafka Streams:
Type 1: Persistent (RocksDB on disk)
Type 2: In-memory (HashMap in JVM heap)
Backing:
Every store has a changelog topic by default
Changelog is compacted (only latest per key)
Partitioning:
Each task owns its own partition of the store
3 partitions = 3 state store instances
How does the persistent RocksDB store work?
The persistent store uses embedded RocksDB, a high-performance key-value database that writes to the local filesystem. Data is stored in SST (Sorted String Table) files on disk. RocksDB can hold datasets much larger than available memory because it reads from disk when needed.
When Kafka Streams starts, RocksDB opens the existing data files from the state directory (state.dir, default /tmp/kafka-streams). If the files are intact, no recovery from the changelog is needed. If the files are missing (new instance or deleted state), the store is rebuilt from the changelog topic.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.state.Stores;
public class PersistentStoreExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Aggregate with persistent RocksDB store (default)
KTable<String, String> balances = transactions
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.aggregate(
() -> "0",
(key, value, aggregate) -> {
String[] parts = value.split("\\|");
long amount = Long.parseLong(parts[2]);
long balance = Long.parseLong(aggregate);
if ("DEPOSIT".equals(parts[1])) {
balance += amount;
} else {
balance -= amount;
}
return String.valueOf(balance);
},
// Persistent RocksDB store (this is the default)
Materialized.as("balance-store")
);
balances.toStream().to("account-balances");
Topology topology = builder.build();
System.out.println("Persistent store (RocksDB):");
System.out.println(" Store name: balance-store");
System.out.println(" Storage: disk (SST files)");
System.out.println(" Location: /tmp/kafka-streams/<app-id>/<task-id>/");
System.out.println(" Survives restart: YES (if state.dir is intact)");
}
}
Expected output:
Persistent store (RocksDB):
Store name: balance-store
Storage: disk (SST files)
Location: /tmp/kafka-streams/<app-id>/<task-id>/
Survives restart: YES (if state.dir is intact)
How does the in-memory store work?
The in-memory store uses a Java TreeMap (for sorted access) or HashMap stored entirely in JVM heap memory. It is faster than RocksDB because there is no disk I/O, but all data is lost when the application stops. On restart, the entire store must be rebuilt from the changelog topic.
Use in-memory stores for small reference datasets like exchange rates, branch configurations, or currency codes where the rebuild time is acceptable.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.state.Stores;
import org.apache.kafka.common.serialization.Serdes;
public class InMemoryStoreExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Read exchange rates into an in-memory KTable
KTable<String, String> exchangeRates = builder.table(
"exchange-rates",
Materialized.<String, String>as(
Stores.inMemoryKeyValueStore("rates-store")
).withKeySerde(Serdes.String())
.withValueSerde(Serdes.String())
);
exchangeRates.toStream().foreach((pair, rate) ->
System.out.println("Rate: " + pair + " = " + rate));
Topology topology = builder.build();
System.out.println("In-memory store:");
System.out.println(" Store name: rates-store");
System.out.println(" Storage: JVM heap (TreeMap)");
System.out.println(" Survives restart: NO (rebuilt from changelog)");
System.out.println(" Best for: small datasets < 100K entries");
}
}
Expected output:
Rate: INR_USD = 0.012
Rate: INR_EUR = 0.011
Rate: INR_GBP = 0.0095
In-memory store:
Store name: rates-store
Storage: JVM heap (TreeMap)
Survives restart: NO (rebuilt from changelog)
Best for: small datasets < 100K entries
How do persistent and in-memory stores compare?
| Feature | Persistent (RocksDB) | In-Memory |
|---|---|---|
| Storage medium | Disk (SST files) | JVM heap (TreeMap) |
| Max dataset size | Larger than memory | Limited by heap size |
| Read latency | Microseconds (cached) to milliseconds (disk) | Nanoseconds (always in memory) |
| Write latency | Microseconds (write to WAL + memtable) | Nanoseconds |
| Survives restart | Yes, if state.dir is intact | No, rebuilt from changelog |
| Recovery on clean restart | Instant (opens existing files) | Full rebuild from changelog |
| Recovery after rebalance | Restore from changelog (or standby) | Full rebuild from changelog |
| Memory usage | Block cache (configurable, default 50MB) | Entire dataset in heap |
| Banking use case | Account balances (50M accounts) | Exchange rates (200 pairs) |
| Default in Kafka Streams | Yes | No (must specify explicitly) |
package com.bank.streams;
public class PersistentVsInMemoryComparison {
public static void main(String[] args) {
System.out.println("=== Persistent (RocksDB) ===");
System.out.println(" 50M accounts, 500 bytes each = ~25 GB on disk");
System.out.println(" Block cache in memory: 50 MB (configurable)");
System.out.println(" Restart: opens existing SST files instantly");
System.out.println();
System.out.println("=== In-Memory ===");
System.out.println(" 200 exchange rate pairs, 50 bytes each = ~10 KB");
System.out.println(" All in JVM heap");
System.out.println(" Restart: rebuild 200 entries from changelog in < 1 second");
System.out.println();
System.out.println("Rule: Use RocksDB (default) for large datasets.");
System.out.println("Use in-memory only for small, fast-changing reference data.");
}
}
Expected output:
=== Persistent (RocksDB) ===
50M accounts, 500 bytes each = ~25 GB on disk
Block cache in memory: 50 MB (configurable)
Restart: opens existing SST files instantly
=== In-Memory ===
200 exchange rate pairs, 50 bytes each = ~10 KB
All in JVM heap
Restart: rebuild 200 entries from changelog in < 1 second
Rule: Use RocksDB (default) for large datasets.
Use in-memory only for small, fast-changing reference data.
What are changelog topics and how does state recovery work?
Every state store write is also written to a changelog topic in Kafka. The changelog topic is named {application.id}-{store-name}-changelog. It uses cleanup.policy=compact so only the latest value per key is retained, keeping the topic size bounded.
When a Kafka Streams instance loses its state (crash, rebalance, or deleted state directory), it replays the changelog topic from the beginning to rebuild the state store. The recovery time depends on the number of unique keys in the changelog.
package com.bank.streams;
public class ChangelogRecoveryExample {
public static void main(String[] args) {
System.out.println("Changelog topic naming:");
System.out.println(" Application ID: txn-analytics-service");
System.out.println(" Store name: balance-store");
System.out.println(" Changelog: txn-analytics-service-balance-store-changelog");
System.out.println();
System.out.println("Recovery scenario:");
System.out.println(" 1. App crashes, state directory deleted");
System.out.println(" 2. App restarts, state store is empty");
System.out.println(" 3. Kafka Streams reads changelog from beginning");
System.out.println(" 4. Each record restores one key-value pair");
System.out.println(" 5. Store fully rebuilt, processing resumes");
System.out.println();
System.out.println("Recovery time for 10M accounts:");
System.out.println(" Changelog size: ~10M records (compacted)");
System.out.println(" Estimated time: 2-5 minutes (depends on network)");
}
}
Expected output:
Changelog topic naming:
Application ID: txn-analytics-service
Store name: balance-store
Changelog: txn-analytics-service-balance-store-changelog
Recovery scenario:
1. App crashes, state directory deleted
2. App restarts, state store is empty
3. Kafka Streams reads changelog from beginning
4. Each record restores one key-value pair
5. Store fully rebuilt, processing resumes
Recovery time for 10M accounts:
Changelog size: ~10M records (compacted)
Estimated time: 2-5 minutes (depends on network)
What are interactive queries?
Interactive queries let you read from a state store without consuming the output topic. You call KafkaStreams.store() with the store name and type, and get a read-only view. This turns your Kafka Streams application into a lightweight key-value data service.
At National Bank of India, Rajesh exposes the account balance state store through a REST API. The branch manager queries the current balance for SAV2001 without consuming the account-balances topic.
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StoreQueryParameters;
import org.apache.kafka.streams.state.QueryableStoreTypes;
import org.apache.kafka.streams.state.ReadOnlyKeyValueStore;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
public class InteractiveQueryExample {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"balance-query-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
// Assume BalanceLookupTopology.build() creates a KTable
// with Materialized.as("balance-store")
KafkaStreams streams = new KafkaStreams(
BalanceLookupTopology.build(), props);
streams.start();
// Wait for the app to be in RUNNING state
Thread.sleep(5000);
// Query the state store directly
ReadOnlyKeyValueStore<String, String> balanceStore =
streams.store(
StoreQueryParameters.fromNameAndType(
"balance-store",
QueryableStoreTypes.keyValueStore()
)
);
// Lookup balance for SAV2001
String balance = balanceStore.get("SAV2001");
System.out.println("Balance for SAV2001: " + balance + " INR");
// Lookup balance for CUR3001
String balance2 = balanceStore.get("CUR3001");
System.out.println("Balance for CUR3001: " + balance2 + " INR");
// Key not found
String balance3 = balanceStore.get("SAV9999");
System.out.println("Balance for SAV9999: " + balance3);
streams.close();
}
}
Expected output:
Balance for SAV2001: 290000 INR
Balance for CUR3001: 500000 INR
Balance for SAV9999: null
What are standby replicas?
Standby replicas are warm copies of state stores maintained on other application instances. When you set num.standby.replicas=1, Kafka Streams assigns one replica task to another instance that continuously reads the changelog topic and keeps its local state store up to date.
When a rebalance happens and a partition moves to a new instance, the new owner can use the standby replica instead of rebuilding from scratch. This reduces recovery time from minutes to seconds.
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;
public class StandbyReplicaConfig {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"balance-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
// Enable 1 standby replica per state store partition
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
System.out.println("Standby replica configuration:");
System.out.println(" num.standby.replicas = 1");
System.out.println();
System.out.println("How it works:");
System.out.println(" Instance A: owns partition 0, active store");
System.out.println(" Instance B: standby for partition 0, reads changelog");
System.out.println();
System.out.println("When Instance A fails:");
System.out.println(" Instance B already has the state store up to date");
System.out.println(" Recovery time: seconds instead of minutes");
System.out.println();
System.out.println("Cost: double the disk usage and changelog read traffic");
}
}
Expected output:
Standby replica configuration:
num.standby.replicas = 1
How it works:
Instance A: owns partition 0, active store
Instance B: standby for partition 0, reads changelog
When Instance A fails:
Instance B already has the state store up to date
Recovery time: seconds instead of minutes
Cost: double the disk usage and changelog read traffic
What is a complete banking project using state stores?
This project is an Account Balance Lookup Service for National Bank of India. It reads transactions, aggregates balances into a named state store, and exposes the store for interactive queries.
Project structure
balance-lookup-service/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── BalanceLookupApp.java
├── BalanceLookupTopology.java
├── BankTransaction.java
└── BalanceLookupTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>balance-lookup-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
BankTransaction.java
package com.bank.streams;
import java.math.BigDecimal;
public class BankTransaction {
private String accountId;
private String type;
private BigDecimal amount;
private String description;
public BankTransaction() {}
public BankTransaction(String accountId, String type,
BigDecimal amount, String description) {
this.accountId = accountId;
this.type = type;
this.amount = amount;
this.description = description;
}
public String getAccountId() { return accountId; }
public String getType() { return type; }
public BigDecimal getAmount() { return amount; }
public String getDescription() { return description; }
@Override
public String toString() {
return accountId + "|" + type + "|" + amount + "|" + description;
}
public static BankTransaction fromString(String value) {
String[] parts = value.split("\\|");
return new BankTransaction(
parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
BalanceLookupTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.math.BigDecimal;
public class BalanceLookupTopology {
public static final String BALANCE_STORE = "balance-store";
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("bank-transactions");
// Aggregate running balance into a named persistent state store
KTable<String, String> balances = transactions
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.aggregate(
() -> "0",
(accountId, txnValue, currentBalance) -> {
BankTransaction txn = BankTransaction.fromString(txnValue);
BigDecimal balance = new BigDecimal(currentBalance);
if ("DEPOSIT".equals(txn.getType())) {
balance = balance.add(txn.getAmount());
} else if ("WITHDRAW".equals(txn.getType())) {
balance = balance.subtract(txn.getAmount());
}
return balance.toPlainString();
},
Materialized.as(BALANCE_STORE)
);
// Also write to output topic for downstream consumers
balances.toStream().to("account-balances");
return builder.build();
}
}
BalanceLookupApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.StoreQueryParameters;
import org.apache.kafka.streams.state.QueryableStoreTypes;
import org.apache.kafka.streams.state.ReadOnlyKeyValueStore;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.Scanner;
import java.util.concurrent.CountDownLatch;
public class BalanceLookupApp {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"balance-lookup-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
KafkaStreams streams = new KafkaStreams(
BalanceLookupTopology.build(), props);
streams.start();
System.out.println("Balance lookup service started");
System.out.println(" Store: " + BalanceLookupTopology.BALANCE_STORE);
System.out.println(" Standby replicas: 1");
System.out.println(" Interactive queries: enabled");
// Wait for RUNNING state
Thread.sleep(5000);
// Interactive query: lookup balance by account ID
ReadOnlyKeyValueStore<String, String> store = streams.store(
StoreQueryParameters.fromNameAndType(
BalanceLookupTopology.BALANCE_STORE,
QueryableStoreTypes.keyValueStore()
)
);
System.out.println("\nLookup SAV2001: " + store.get("SAV2001") + " INR");
System.out.println("Lookup CUR3001: " + store.get("CUR3001") + " INR");
streams.close();
}
}
Expected output:
Balance lookup service started
Store: balance-store
Standby replicas: 1
Interactive queries: enabled
Lookup SAV2001: 290000 INR
Lookup CUR3001: 500000 INR
BalanceLookupTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class BalanceLookupTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(BalanceLookupTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"bank-transactions", new StringSerializer(), new StringSerializer());
input.pipeInput("SAV2001", "SAV2001|DEPOSIT|250000|salary credit");
input.pipeInput("SAV2001", "SAV2001|WITHDRAW|10000|atm withdrawal");
input.pipeInput("CUR3001", "CUR3001|DEPOSIT|500000|vendor payment");
// Query the state store directly via TopologyTestDriver
KeyValueStore<String, String> store =
driver.getKeyValueStore(BalanceLookupTopology.BALANCE_STORE);
System.out.println("=== State Store Contents ===");
System.out.println("SAV2001: " + store.get("SAV2001") + " INR");
System.out.println("CUR3001: " + store.get("CUR3001") + " INR");
System.out.println("SAV9999: " + store.get("SAV9999"));
System.out.println();
System.out.println("Store size: " + store.approximateNumEntries() + " entries");
}
}
}
Expected output:
=== State Store Contents ===
SAV2001: 240000 INR
CUR3001: 500000 INR
SAV9999: null
Store size: 2 entries
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics
kafka-topics.sh --create --topic bank-transactions --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic account-balances --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build and run
cd balance-lookup-service
mvn clean package
java -cp target/balance-lookup-service-1.0.0.jar com.bank.streams.BalanceLookupApp
# Step 4: Produce transactions
kafka-console-producer.sh --topic bank-transactions --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|DEPOSIT|250000|salary credit
# SAV2001:SAV2001|WITHDRAW|10000|atm withdrawal
# CUR3001:CUR3001|DEPOSIT|500000|vendor payment
What are common mistakes with state stores?
The most common mistake is not configuring the state directory for production.
Wrong: using default state.dir in production
package com.bank.streams;
public class WrongDefaultStateDir {
public static void main(String[] args) {
System.out.println("WRONG: Using default state.dir in production");
System.out.println();
System.out.println("Default: state.dir = /tmp/kafka-streams");
System.out.println();
System.out.println("Problems:");
System.out.println(" 1. /tmp is cleared on OS reboot");
System.out.println(" 2. State store files are lost");
System.out.println(" 3. Full changelog replay needed on every reboot");
System.out.println(" 4. For 50M accounts, replay takes 5-10 minutes");
}
}
Expected output:
WRONG: Using default state.dir in production
Default: state.dir = /tmp/kafka-streams
Problems:
1. /tmp is cleared on OS reboot
2. State store files are lost
3. Full changelog replay needed on every reboot
4. For 50M accounts, replay takes 5-10 minutes
Correct: configuring state.dir on a persistent volume
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;
public class CorrectStateDir {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "balance-service");
// Use a persistent volume for state storage
props.put(StreamsConfig.STATE_DIR_CONFIG, "/data/kafka-streams");
System.out.println("CORRECT: state.dir = /data/kafka-streams");
System.out.println();
System.out.println("Benefits:");
System.out.println(" 1. Survives OS reboot");
System.out.println(" 2. RocksDB opens existing files on restart");
System.out.println(" 3. No changelog replay needed");
System.out.println(" 4. Instant recovery");
System.out.println();
System.out.println("In Kubernetes: mount a PersistentVolumeClaim at /data/kafka-streams");
}
}
Expected output:
CORRECT: state.dir = /data/kafka-streams
Benefits:
1. Survives OS reboot
2. RocksDB opens existing files on restart
3. No changelog replay needed
4. Instant recovery
In Kubernetes: mount a PersistentVolumeClaim at /data/kafka-streams
What should a senior or principal engineer know?
At the senior level, you should understand RocksDB tuning, changelog management, and operational concerns for state stores in production.
- RocksDB block cache: The default block cache is 50 MB per state store. For large stores, increase it to 256 MB or more. Set via StreamsConfig with a custom RocksDBConfigSetter.
- Write buffer size: RocksDB writes go to an in-memory write buffer (memtable) before flushing to disk. Increasing write_buffer_size reduces write amplification but uses more memory.
- Changelog retention: Changelog topics are compacted by default. If you also add a time-based retention (e.g., 7 days), records older than 7 days are deleted even if they are the latest for a key. This can cause data loss during recovery. Keep changelog topics as compact-only.
- State store sizing formula: Total disk usage per store = (number of unique keys) x (average key size + average value size) x 1.5 (RocksDB overhead). For 50M accounts with 500-byte values, expect about 37.5 GB.
- Monitoring: Expose JMX metrics for state store size (rocksdb.estimate-num-keys), block cache hit rate (rocksdb.block-cache-hit-ratio), and changelog consumer lag.
- Disable changelog for read-only stores: If you use a GlobalKTable for reference data that is already replicated in Kafka, you can disable the changelog with Materialized.withLoggingDisabled() to save storage.
package com.bank.streams;
public class SeniorStateStoreKnowledge {
public static void main(String[] args) {
System.out.println("Senior checklist for state stores:");
System.out.println(" 1. Set state.dir to persistent volume, not /tmp");
System.out.println(" 2. Tune RocksDB block cache (default 50 MB per store)");
System.out.println(" 3. Keep changelog as compact-only, no time retention");
System.out.println(" 4. Enable standby replicas for fast failover");
System.out.println(" 5. Monitor: estimate-num-keys, block-cache-hit-ratio, changelog-lag");
System.out.println(" 6. Size formula: keys x (key + value size) x 1.5 overhead");
System.out.println(" 7. Disable changelog for GlobalKTable reference stores");
}
}
Expected output:
Senior checklist for state stores:
1. Set state.dir to persistent volume, not /tmp
2. Tune RocksDB block cache (default 50 MB per store)
3. Keep changelog as compact-only, no time retention
4. Enable standby replicas for fast failover
5. Monitor: estimate-num-keys, block-cache-hit-ratio, changelog-lag
6. Size formula: keys x (key + value size) x 1.5 overhead
7. Disable changelog for GlobalKTable reference stores
What are common interview follow-up questions?
- Can you use a state store without an aggregation? (Yes. You can add a custom state store to a topology using builder.addStateStore() and access it in a Processor or Transformer. This is useful for custom lookup tables or caches.)
- What happens if the state store runs out of disk space? (RocksDB throws an IOException and the stream thread dies. The task enters ERROR state. You must free disk space or increase the volume before restarting.)
- Can two applications share the same state store? (No. Each application.id has its own state stores and changelog topics. State stores are local to the application instance and partition.)
- How does Kafka Streams decide where to restore state after a rebalance? (It uses the partition assignment strategy. If a partition moves from instance A to instance B, instance B checks if it has a standby replica. If yes, it promotes the standby. If no, it replays the changelog.)
- Can you query a state store from a different machine? (Not directly. Interactive queries are local to the instance that owns the partition. For cross-instance queries, you need a discovery mechanism. Kafka Streams provides streams.metadataForKey() to find which instance hosts a specific key.)
- What is the difference between Materialized.as("name") and Materialized.as(Stores.persistentKeyValueStore("name"))? (The first uses the default store type (persistent RocksDB). The second explicitly specifies persistent. They are equivalent. Use Stores.inMemoryKeyValueStore() if you want in-memory.)
Visual Overview
State Store Architecture
==========================
┌─────────────────────────────────────────┐
│ Kafka Streams Application │
│ │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Task 0 │ │ Task 1 │ │
│ │ Partition 0 │ │ Partition 1 │ │
│ │ │ │ │ │
│ │ ┌─────────┐ │ │ ┌─────────┐ │ │
│ │ │ State │ │ │ │ State │ │ │
│ │ │ Store 0 │ │ │ │ Store 1 │ │ │
│ │ │(RocksDB)│ │ │ │(RocksDB)│ │ │
│ │ └────┬────┘ │ │ └────┬────┘ │ │
│ └──────┼──────┘ └──────┼──────┘ │
│ │ │ │
└─────────┼─────────────────┼─────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────┐
│ Changelog Topic (Kafka) │
│ app-id-store-name-changelog │
│ cleanup.policy = compact │
│ │
│ Partition 0: SAV2001=290000 │
│ Partition 1: CUR3001=500000 │
└─────────────────────────────────────────┘
Recovery Flow
===============
Normal Restart (state.dir intact):
┌──────────┐ ┌──────────────┐
│ App │───→│ Open existing│───→ Ready (instant)
│ starts │ │ RocksDB files│
└──────────┘ └──────────────┘
State Lost (state.dir deleted):
┌──────────┐ ┌──────────────┐ ┌────────────┐
│ App │───→│ Empty store │───→│ Replay │───→ Ready
│ starts │ │ detected │ │ changelog │ (minutes)
└──────────┘ └──────────────┘ └────────────┘
With Standby Replica:
┌──────────┐ ┌──────────────┐
│ Rebalance│───→│ Promote │───→ Ready (seconds)
│ detected │ │ standby │
└──────────┘ └──────────────┘
Key Takeaway
State stores are the foundation of stateful processing in Kafka Streams. Every aggregation, every KTable, and every join uses a state store to hold accumulated data. Understanding how they work is essential for building reliable stream processing applications.
The persistent RocksDB store is the default and the right choice for most production workloads. It writes data to disk as SST files, handles datasets much larger than available memory, and survives application restarts when the state directory is on a persistent volume. The in-memory store is faster but loses all data on shutdown, making it suitable only for small reference datasets like exchange rates or branch configurations that can be rebuilt quickly from the changelog.
Every state store is backed by a changelog topic in Kafka. This topic is compacted, keeping only the latest value per key. When a state store is lost (crash, rebalance, or deleted state directory), it is rebuilt by replaying the changelog from the beginning. Standby replicas keep a warm copy on another instance, reducing recovery time from minutes to seconds. At National Bank of India, Rajesh configures one standby replica for the balance store so that branch managers never wait more than a few seconds during failover.
Interactive queries turn a Kafka Streams application into a lightweight data service. By calling KafkaStreams.store() with the store name, you get a read-only view of the state store. The branch manager can query the current balance for SAV2001 without consuming the output topic.
At the senior level, the critical concerns are: set state.dir to a persistent volume (not /tmp), tune RocksDB block cache for your dataset size, keep changelog topics as compact-only without time-based retention, and monitor state store size and changelog consumer lag via JMX.
Remember one sentence: a state store is local key-value storage backed by a changelog topic for recovery, and the choice between persistent RocksDB and in-memory determines whether your data survives a restart or must be rebuilt.
---