Definition
- KStream is an unbounded, append-only stream of records where every record with the same key is kept as a separate event.
- KTable is a changelog stream where each new record with the same key replaces the previous one, representing the latest state.
- Event is a fact that happened at a point in time; it is immutable and should not be overwritten.
- State is the current value derived from a sequence of events; only the latest value is meaningful.
- Compacted topic is a Kafka topic where the broker deletes older records with the same key, keeping only the latest, which is the natural storage model for a KTable.
Think of National Bank of India's daily operations. The transaction register at the teller counter is a KStream: every deposit, withdrawal, and transfer is written as a new line, and no entry is ever erased or replaced. The register grows with every transaction. The balance display board behind the manager is a KTable: it shows one row per account with the current balance. When a new transaction arrives, the balance for that account is updated in place. The register keeps the full history; the balance board shows only the current state.
Understanding the Concepts
What is KStream?
KStream represents an unbounded, continuously updating stream of records. Every record is an independent, immutable event. When multiple records arrive with the same key, all of them are kept in the stream. No record replaces another.
In a banking application, a KStream is the natural choice for a transaction log. Every deposit, withdrawal, and transfer is a separate event. Rajesh, a developer at National Bank of India, uses a KStream to process payment events because the bank needs a complete history of every transaction.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class KStreamBasicExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// KStream reads every record as an independent event
KStream<String, String> transactions = builder.stream("bank-transactions");
// Three records for SAV2001 are all processed separately
transactions.foreach((accountId, txn) ->
System.out.println("Event: account=" + accountId + ", txn=" + txn));
Topology topology = builder.build();
System.out.println("KStream topology:");
System.out.println(topology.describe());
}
}
Expected output (when three records arrive for SAV2001):
Event: account=SAV2001, txn=DEPOSIT 50000
Event: account=SAV2001, txn=WITHDRAW 10000
Event: account=SAV2001, txn=DEPOSIT 25000
All 3 records processed. No record replaced another.
What is KTable?
KTable represents a changelog stream where each record is an update to the value of a key. When a new record arrives with the same key, it replaces the previous value. A KTable always holds the latest value per key, backed by a local state store.
In banking, the current account balance is a KTable. When Priya deposits 50,000 rupees into SAV2001, the balance changes from the old value to the new value. The bank does not display every historical balance on the board, only the current one.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
public class KTableBasicExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// KTable keeps only the latest value per key
KTable<String, String> balances = builder.table("account-balances",
Materialized.as("balance-store"));
// Convert to stream to observe changes
balances.toStream().foreach((accountId, balance) ->
System.out.println("Current balance: account=" + accountId
+ ", balance=" + balance));
Topology topology = builder.build();
System.out.println("KTable topology:");
System.out.println(topology.describe());
}
}
Expected output (after SAV2001 receives 3 balance updates: 50000, 90000, 65000):
Current balance: account=SAV2001, balance=50000
Current balance: account=SAV2001, balance=90000
Current balance: account=SAV2001, balance=65000
KTable emits each change event, but the state store holds only the latest: 65000
How do KStream and KTable differ?
The core difference is how records with the same key are handled. KStream keeps all records; KTable keeps only the latest per key. This affects memory usage, processing semantics, and the type of Kafka topic that backs each abstraction.
| Feature | KStream | KTable |
|---|---|---|
| Record model | Every record is an independent event | Each record updates the previous value for that key |
| Duplicate keys | All records retained | Latest value per key retained |
| Backed by | Regular Kafka topic | Compacted Kafka topic or aggregation result |
| State store | No local state for stateless operations | Local state store (RocksDB or in-memory) |
| Read method | builder.stream(topic) | builder.table(topic) |
| Memory usage | No extra memory (records pass through) | Memory proportional to unique keys |
| Banking example | Transaction log | Account balance ledger |
| Conversion | .toTable() creates a KTable | .toStream() creates a KStream |
package com.bank.streams;
public class KStreamVsKTableComparison {
public static void main(String[] args) {
System.out.println("=== KStream behavior ===");
System.out.println("Record 1: SAV2001 -> DEPOSIT 50000 (kept)");
System.out.println("Record 2: SAV2001 -> WITHDRAW 10000 (kept)");
System.out.println("Record 3: SAV2001 -> DEPOSIT 25000 (kept)");
System.out.println("Total records in stream: 3");
System.out.println();
System.out.println("=== KTable behavior ===");
System.out.println("Record 1: SAV2001 -> 50000 (stored)");
System.out.println("Record 2: SAV2001 -> 40000 (replaces 50000)");
System.out.println("Record 3: SAV2001 -> 65000 (replaces 40000)");
System.out.println("Total records in table: 1 (latest value: 65000)");
}
}
Expected output:
=== KStream behavior ===
Record 1: SAV2001 -> DEPOSIT 50000 (kept)
Record 2: SAV2001 -> WITHDRAW 10000 (kept)
Record 3: SAV2001 -> DEPOSIT 25000 (kept)
Total records in stream: 3
=== KTable behavior ===
Record 1: SAV2001 -> 50000 (stored)
Record 2: SAV2001 -> 40000 (replaces 50000)
Record 3: SAV2001 -> 65000 (replaces 40000)
Total records in table: 1 (latest value: 65000)
When should you use KStream in a banking application?
Use KStream when every event matters independently and you need the complete history. The three criteria are: (1) events are immutable facts, (2) multiple records with the same key are all meaningful, and (3) you process or forward each event individually.
- Banking use cases for KStream:
- Payment transactions: every deposit, withdrawal, and transfer is a separate event that must be recorded
- Fraud alerts: every suspicious activity triggers an independent alert
- Audit trail: regulatory compliance requires a complete, unmodified event history
- Notification triggers: every account event sends a separate SMS or email to Priya
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
public class KStreamBankingUseCases {
private static final BigDecimal FRAUD_THRESHOLD = new BigDecimal("500000");
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> transactions = builder.stream("payment-transactions");
// Use case 1: Route high-value transactions to fraud review
KStream<String, String> suspicious = transactions
.filter((accountId, txn) -> {
BigDecimal amount = new BigDecimal(txn.split("\\|")[2]);
return amount.compareTo(FRAUD_THRESHOLD) > 0;
});
suspicious.to("fraud-review-queue");
// Use case 2: Forward every transaction to audit log
transactions.to("audit-log");
Topology topology = builder.build();
System.out.println("KStream banking use cases:");
System.out.println(" 1. Fraud detection: filter transactions > 500000 INR");
System.out.println(" 2. Audit log: forward all transactions unchanged");
System.out.println("Both paths need every event. KStream is the right choice.");
}
}
Expected output:
KStream banking use cases:
1. Fraud detection: filter transactions > 500000 INR
2. Audit log: forward all transactions unchanged
Both paths need every event. KStream is the right choice.
When should you use KTable in a banking application?
Use KTable when only the latest value per key matters and previous values can be discarded. The three criteria are: (1) each record is an update, not an independent fact, (2) only the most recent value for a key is useful, and (3) you need to look up the current state by key.
- Banking use cases for KTable:
- Account balances: only the current balance matters for loan eligibility or overdraft checks
- Customer profiles: the latest name, address, and KYC status for Priya
- Exchange rates: only the current INR/USD rate is relevant for conversions
- Branch configuration: current operating hours and service availability for Mumbai Branch
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
public class KTableBankingUseCases {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Current account balances (latest per account)
KTable<String, String> balances = builder.table("account-balances",
Materialized.as("balance-store"));
// Customer profiles (latest KYC status per customer)
KTable<String, String> profiles = builder.table("customer-profiles",
Materialized.as("profile-store"));
// Exchange rates (latest rate per currency pair)
KTable<String, String> rates = builder.table("exchange-rates",
Materialized.as("rates-store"));
System.out.println("KTable banking use cases:");
System.out.println(" 1. Account balances: SAV2001 -> 245000");
System.out.println(" 2. Customer profiles: SAV2001 -> Priya Sharma|KYC_VERIFIED");
System.out.println(" 3. Exchange rates: INR_USD -> 0.012");
System.out.println("All three need only the latest value. KTable is the right choice.");
}
}
Expected output:
KTable banking use cases:
1. Account balances: SAV2001 -> 245000
2. Customer profiles: SAV2001 -> Priya Sharma|KYC_VERIFIED
3. Exchange rates: INR_USD -> 0.012
All three need only the latest value. KTable is the right choice.
How do you convert between KStream and KTable?
Kafka Streams provides two conversion methods. KStream.toTable() converts a stream into a table by keeping only the latest value per key. KTable.toStream() converts a table back into a stream, emitting a record for every state change.
The most common use case for toTable() is when you receive events on a regular topic but need lookup semantics. The most common use case for toStream() is when you need to write a KTable's changes to an output topic using the .to() method, which is available only on KStream.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.Topology;
public class StreamTableConversion {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Start with a KStream of balance updates
KStream<String, String> balanceUpdates = builder.stream("balance-updates");
// Convert KStream to KTable: only the latest balance per account is kept
KTable<String, String> currentBalances = balanceUpdates.toTable();
// Convert KTable back to KStream: emits every change as an event
KStream<String, String> changeLog = currentBalances.toStream();
// Write the change log to an output topic
changeLog.to("balance-changelog");
Topology topology = builder.build();
System.out.println("Conversion topology:");
System.out.println(topology.describe());
}
}
Expected output:
Conversion topology:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [balance-updates])
--> KSTREAM-TOTABLE-0000000001
Processor: KSTREAM-TOTABLE-0000000001 (stores: [...])
--> KTABLE-TOSTREAM-0000000002
Processor: KTABLE-TOSTREAM-0000000002 (stores: [])
--> KSTREAM-SINK-0000000003
Sink: KSTREAM-SINK-0000000003 (topic: balance-changelog)
<-- KTABLE-TOSTREAM-0000000002
How do you use KStream and KTable together in one topology?
The most powerful pattern is joining a KStream of events with a KTable of reference data. This enriches each event with the current state without storing every event. At National Bank of India, Rajesh joins the transaction stream (KStream) with the customer profile table (KTable) to add the customer name and branch to each transaction before writing to the enriched output topic.
The join works by key: for each record in the KStream, Kafka Streams looks up the current value in the KTable with the same key and produces a joined result.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.Topology;
public class StreamTableJoinExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// KStream: payment transactions (key = accountId)
KStream<String, String> transactions = builder.stream("payment-transactions");
// KTable: customer profiles (key = accountId)
KTable<String, String> profiles = builder.table("customer-profiles");
// Join each transaction with the customer's current profile
KStream<String, String> enriched = transactions.join(
profiles,
(txnValue, profileValue) -> txnValue + "|CUSTOMER:" + profileValue
);
enriched.to("enriched-transactions");
Topology topology = builder.build();
System.out.println("KStream-KTable join topology:");
System.out.println(" Input stream: payment-transactions");
System.out.println(" Lookup table: customer-profiles");
System.out.println(" Output stream: enriched-transactions");
System.out.println("Each transaction is enriched with the customer's current profile.");
}
}
Expected output:
KStream-KTable join topology:
Input stream: payment-transactions
Lookup table: customer-profiles
Output stream: enriched-transactions
Each transaction is enriched with the customer's current profile.
Example join result:
Transaction: SAV2001|DEPOSIT|50000|neft transfer
Profile: Priya Sharma|Mumbai Branch
Enriched: SAV2001|DEPOSIT|50000|neft transfer|CUSTOMER:Priya Sharma|Mumbai Branch
What is a complete banking project using KStream and KTable?
This project is an Account Activity Classifier for National Bank of India. It reads account events from a single topic and processes them through two paths: a KStream path filters high-value transactions for real-time alerts, and a KTable path aggregates a running balance per account. A junior developer can build and run this project from start to finish.
Project structure
account-activity-classifier/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── AccountActivityApp.java
├── AccountActivityTopology.java
├── AccountEvent.java
└── AccountActivityTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>account-activity-classifiersyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
AccountEvent.java
package com.bank.streams;
import java.math.BigDecimal;
public class AccountEvent {
private String accountId;
private String type;
private BigDecimal amount;
private String description;
public AccountEvent() {}
public AccountEvent(String accountId, String type,
BigDecimal amount, String description) {
this.accountId = accountId;
this.type = type;
this.amount = amount;
this.description = description;
}
public String getAccountId() { return accountId; }
public void setAccountId(String accountId) { this.accountId = accountId; }
public String getType() { return type; }
public void setType(String type) { this.type = type; }
public BigDecimal getAmount() { return amount; }
public void setAmount(BigDecimal amount) { this.amount = amount; }
public String getDescription() { return description; }
public void setDescription(String description) { this.description = description; }
@Override
public String toString() {
return accountId + "|" + type + "|" + amount + "|" + description;
}
public static AccountEvent fromString(String value) {
String[] parts = value.split("\\|");
return new AccountEvent(
parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
AccountActivityTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.math.BigDecimal;
public class AccountActivityTopology {
private static final BigDecimal ALERT_THRESHOLD = new BigDecimal("100000");
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// KStream path: read every event as an independent fact
KStream<String, String> events = builder.stream("account-events");
// Filter high-value events and write to alerts topic
events
.filter((key, value) -> {
AccountEvent event = AccountEvent.fromString(value);
return event.getAmount().compareTo(ALERT_THRESHOLD) > 0;
})
.to("high-value-alerts");
// KTable path: aggregate running balance per account
KTable<String, String> balances = events
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.aggregate(
() -> "0",
(accountId, eventValue, currentBalance) -> {
AccountEvent event = AccountEvent.fromString(eventValue);
BigDecimal balance = new BigDecimal(currentBalance);
if ("DEPOSIT".equals(event.getType())) {
balance = balance.add(event.getAmount());
} else if ("WITHDRAW".equals(event.getType())) {
balance = balance.subtract(event.getAmount());
}
return balance.toPlainString();
},
Materialized.as("account-balances-store")
);
// Write the KTable changes to an output topic
balances.toStream().to("account-balances");
return builder.build();
}
}
AccountActivityApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class AccountActivityApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"account-activity-classifier");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
AccountActivityTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Account activity classifier shut down");
}));
try {
streams.start();
System.out.println("Account activity classifier started");
System.out.println(" Input topic: account-events");
System.out.println(" Alerts topic: high-value-alerts");
System.out.println(" Balance topic: account-balances");
System.out.println(" Threshold: 100000 INR");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
Expected output (on startup):
Account activity classifier started
Input topic: account-events
Alerts topic: high-value-alerts
Balance topic: account-balances
Threshold: 100000 INR
AccountActivityTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class AccountActivityTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(AccountActivityTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"account-events", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> alerts = driver.createOutputTopic(
"high-value-alerts", new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> balances = driver.createOutputTopic(
"account-balances", new StringDeserializer(), new StringDeserializer());
// Deposit 250000 (above threshold, triggers alert)
input.pipeInput("SAV2001", "SAV2001|DEPOSIT|250000|neft from Priya");
// Withdrawal 5000 (below threshold, no alert)
input.pipeInput("SAV2001", "SAV2001|WITHDRAW|5000|atm withdrawal");
// Deposit 500000 (above threshold, triggers alert)
input.pipeInput("CUR3001", "CUR3001|DEPOSIT|500000|rtgs from vendor");
System.out.println("=== High-value alerts (KStream path) ===");
System.out.println("Alert count: " + alerts.getQueueSize());
System.out.println("Alert 1: " + alerts.readValue());
System.out.println("Alert 2: " + alerts.readValue());
System.out.println("Alerts empty: " + alerts.isEmpty());
System.out.println();
System.out.println("=== Account balances (KTable path) ===");
while (!balances.isEmpty()) {
var record = balances.readKeyValue();
System.out.println("Balance: " + record.key + " = " + record.value);
}
}
}
}
Expected output:
=== High-value alerts (KStream path) ===
Alert count: 2
Alert 1: SAV2001|DEPOSIT|250000|neft from Priya
Alert 2: CUR3001|DEPOSIT|500000|rtgs from vendor
Alerts empty: true
=== Account balances (KTable path) ===
Balance: SAV2001 = 250000
Balance: SAV2001 = 245000
Balance: CUR3001 = 500000
Notice that the 5,000 rupee withdrawal did not appear in alerts (below threshold), but the balance for SAV2001 updated from 250,000 to 245,000. The KStream path and KTable path work independently in the same topology.
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics
kafka-topics.sh --create --topic account-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic high-value-alerts --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic account-balances --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build the project
cd account-activity-classifier
mvn clean package
# Step 4: Run the Kafka Streams app
java -cp target/account-activity-classifier-1.0.0.jar com.bank.streams.AccountActivityApp
# Step 5: Produce test messages (key:value format)
kafka-console-producer.sh --topic account-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|DEPOSIT|250000|neft from Priya
# SAV2001:SAV2001|WITHDRAW|5000|atm withdrawal
# CUR3001:CUR3001|DEPOSIT|500000|rtgs from vendor
# Step 6: Consume alerts
kafka-console-consumer.sh --topic high-value-alerts --from-beginning --bootstrap-server localhost:9092
# Step 7: Consume balances
kafka-console-consumer.sh --topic account-balances --from-beginning --bootstrap-server localhost:9092
What are common mistakes when choosing KStream vs KTable?
The most common mistake is using the wrong abstraction for the data model. This causes either data loss or unnecessary memory consumption.
Wrong: using KTable for an audit log
package com.bank.streams;
public class WrongKTableForAuditLog {
public static void main(String[] args) {
System.out.println("WRONG: Using KTable for audit log");
System.out.println();
System.out.println("Code: KTable<String, String> audit = builder.table(\"audit-events\");");
System.out.println();
System.out.println("What happens:");
System.out.println(" Event 1: SAV2001 -> LOGIN at 09:00 (stored)");
System.out.println(" Event 2: SAV2001 -> TRANSFER at 09:05 (replaces LOGIN)");
System.out.println(" Event 3: SAV2001 -> LOGOUT at 09:10 (replaces TRANSFER)");
System.out.println();
System.out.println("Result: Only LOGOUT is in the table. LOGIN and TRANSFER are lost.");
System.out.println("This violates RBI audit trail requirements.");
}
}
Expected output:
WRONG: Using KTable for audit log
Code: KTable<String, String> audit = builder.table("audit-events");
What happens:
Event 1: SAV2001 -> LOGIN at 09:00 (stored)
Event 2: SAV2001 -> TRANSFER at 09:05 (replaces LOGIN)
Event 3: SAV2001 -> LOGOUT at 09:10 (replaces TRANSFER)
Result: Only LOGOUT is in the table. LOGIN and TRANSFER are lost.
This violates RBI audit trail requirements.
Correct: using KStream for an audit log
package com.bank.streams;
public class CorrectKStreamForAuditLog {
public static void main(String[] args) {
System.out.println("CORRECT: Using KStream for audit log");
System.out.println();
System.out.println("Code: KStream<String, String> audit = builder.stream(\"audit-events\");");
System.out.println();
System.out.println("What happens:");
System.out.println(" Event 1: SAV2001 -> LOGIN at 09:00 (processed)");
System.out.println(" Event 2: SAV2001 -> TRANSFER at 09:05 (processed)");
System.out.println(" Event 3: SAV2001 -> LOGOUT at 09:10 (processed)");
System.out.println();
System.out.println("Result: All 3 events processed and forwarded. Complete audit trail.");
}
}
Expected output:
CORRECT: Using KStream for audit log
Code: KStream<String, String> audit = builder.stream("audit-events");
What happens:
Event 1: SAV2001 -> LOGIN at 09:00 (processed)
Event 2: SAV2001 -> TRANSFER at 09:05 (processed)
Event 3: SAV2001 -> LOGOUT at 09:10 (processed)
Result: All 3 events processed and forwarded. Complete audit trail.
Wrong: using KStream where KTable is needed
package com.bank.streams;
public class WrongKStreamForBalance {
public static void main(String[] args) {
System.out.println("WRONG: Reading balance topic as KStream for lookups");
System.out.println();
System.out.println("Code: KStream<String, String> balances = builder.stream(\"account-balances\");");
System.out.println();
System.out.println("Problem: KStream has no state store. You cannot look up");
System.out.println("the current balance for SAV2001. Every record flows through");
System.out.println("but nothing is stored for later lookups or joins.");
System.out.println();
System.out.println("CORRECT: Read as KTable");
System.out.println("Code: KTable<String, String> balances = builder.table(\"account-balances\");");
System.out.println();
System.out.println("KTable materializes the latest balance in a state store.");
System.out.println("You can join other streams against it or query it directly.");
}
}
Expected output:
WRONG: Reading balance topic as KStream for lookups
Code: KStream<String, String> balances = builder.stream("account-balances");
Problem: KStream has no state store. You cannot look up
the current balance for SAV2001. Every record flows through
but nothing is stored for later lookups or joins.
CORRECT: Read as KTable
Code: KTable<String, String> balances = builder.table("account-balances");
KTable materializes the latest balance in a state store.
You can join other streams against it or query it directly.
What should a senior or principal engineer know?
At the senior level, you should understand the operational implications of choosing KStream vs KTable.
- Memory footprint: A KTable stores one entry per unique key in a local RocksDB state store. If the topic has 10 million unique accounts, the state store holds 10 million entries. A KStream with stateless operations uses no local state.
- Topic compaction: A KTable should be backed by a compacted topic (cleanup.policy=compact). Without compaction, the topic grows without bound and KTable restoration on restart takes longer.
- Repartitioning: If you call KStream.selectKey() and then join with a KTable, Kafka Streams creates an internal repartition topic. This adds network overhead and latency.
- Changelog overhead: Stateful operations on KStream (like groupByKey().aggregate()) create a KTable internally, which is backed by a changelog topic. This changelog topic adds storage and replication costs.
- Interactive queries: A KTable materialized with a named store can be queried via the Interactive Queries API. This turns your Kafka Streams app into a lightweight read-only data store.
- Rebalancing impact: KTable state stores must be restored after a rebalance. More state means longer restoration time. Use standby replicas (num.standby.replicas) to reduce restoration latency.
package com.bank.streams;
public class SeniorKnowledgeKStreamKTable {
public static void main(String[] args) {
System.out.println("Senior checklist for KStream vs KTable:");
System.out.println(" 1. KTable memory = unique keys x avg value size");
System.out.println(" 2. Back KTable topics with cleanup.policy=compact");
System.out.println(" 3. selectKey() before join triggers repartition topic");
System.out.println(" 4. groupByKey().aggregate() creates internal changelog");
System.out.println(" 5. Named stores enable interactive queries via REST");
System.out.println(" 6. Standby replicas reduce state restoration time");
System.out.println(" 7. Monitor: state-store-size, rebalance-latency, changelog-lag");
}
}
Expected output:
Senior checklist for KStream vs KTable:
1. KTable memory = unique keys x avg value size
2. Back KTable topics with cleanup.policy=compact
3. selectKey() before join triggers repartition topic
4. groupByKey().aggregate() creates internal changelog
5. Named stores enable interactive queries via REST
6. Standby replicas reduce state restoration time
7. Monitor: state-store-size, rebalance-latency, changelog-lag
What are common interview follow-up questions?
- Can a KTable have more records than the number of unique keys in the topic? (No. A KTable retains exactly one value per key. If the topic has 1,000 unique keys, the KTable has at most 1,000 entries.)
- What happens if you send a null value for a key to a KTable? (The key is deleted from the KTable. This is called a tombstone. It removes the entry from the state store and the compacted topic.)
- Can you join two KStreams without a window? (No. KStream-KStream joins require a JoinWindows parameter because both sides are unbounded. Without a window, the join would need to buffer all records forever.)
- Does KTable.toStream() replay the entire table? (No. It only emits future changes. If you need the full table state, use interactive queries or read the changelog topic from the beginning.)
- How does KTable handle out-of-order records? (KTable uses the record's offset order, not timestamp order. The record with the highest offset for a key is the current value. Out-of-order timestamps do not affect KTable state.)
- Can you use GlobalKTable instead of KTable? (Yes, but GlobalKTable replicates the entire topic to every instance. Use it for small reference data like branch directories or currency codes, not for large tables like account balances.)
Visual Overview
KStream Flow (All Records Kept)
=================================
┌───────────────────────────┐
│ Topic: bank-transactions │
│ SAV2001 -> DEPOSIT 50000 │
│ SAV2001 -> WITHDRAW 10000│
│ SAV2001 -> DEPOSIT 25000 │
└────────────┬──────────────┘
│ builder.stream()
▼
┌───────────────────────────┐
│ KStream │
│ Record 1: DEPOSIT 50000 │ ← kept
│ Record 2: WITHDRAW 10000 │ ← kept
│ Record 3: DEPOSIT 25000 │ ← kept
│ Total: 3 records │
└───────────────────────────┘
KTable Flow (Latest Per Key)
==============================
┌───────────────────────────┐
│ Topic: account-balances │
│ SAV2001 -> 50000 │
│ SAV2001 -> 90000 │
│ SAV2001 -> 65000 │
└────────────┬──────────────┘
│ builder.table()
▼
┌───────────────────────────┐
│ KTable (State Store) │
│ SAV2001 -> 65000 │ ← only latest
│ Total: 1 entry │
└───────────────────────────┘
Combined Topology
===================
┌───────────────────────────┐
│ Topic: account-events │
└────────────┬──────────────┘
│
┌────┴─────┐
│ KStream │
└────┬─────┘
│
┌────────┴────────┐
│ │
▼ ▼
┌─────────┐ ┌────────────────┐
│ filter │ │ groupByKey() │
│ > 100K │ │ .aggregate() │
└────┬────┘ └───────┬────────┘
│ │
▼ ▼
┌──────────┐ ┌────────────────┐
│ high- │ │ KTable │
│ value- │ │ (balances) │
│ alerts │ └───────┬────────┘
└──────────┘ │ .toStream()
▼
┌────────────────┐
│ account- │
│ balances │
└────────────────┘
Key Takeaway
KStream and KTable are the two fundamental abstractions in Kafka Streams, and choosing between them is the first design decision you make when building a stream processing topology.
KStream treats every record as an independent event. When three deposits arrive for account SAV2001, all three are processed separately. Nothing is overwritten. This is the right model for transaction logs, fraud alerts, audit trails, and notification triggers. At National Bank of India, Rajesh uses a KStream to process every payment event because regulatory compliance requires a complete, unmodified history.
KTable treats records as updates to a key's current state. When three balance updates arrive for SAV2001, only the latest value is kept. Previous values are discarded from the local state store. This is the right model for account balances, customer profiles, exchange rates, and branch configurations. The balance display board behind the bank manager is a KTable: one row per account, always showing the current number.
The most powerful pattern combines both abstractions in a single topology. You read events as a KStream, filter high-value transactions for alerts, and simultaneously aggregate a running balance into a KTable. The KStream path handles real-time event processing; the KTable path maintains current state. You can also join a KStream with a KTable to enrich each event with reference data.
The most common mistake is choosing the wrong abstraction. Using a KTable for an audit log loses events because each new record replaces the previous one for the same key. Using a KStream where you need lookup semantics gives you no state store to query. At the senior level, understand that KTable memory is proportional to unique keys, that compaction is essential for KTable topics, and that standby replicas reduce restoration time after rebalancing.
Remember one sentence: KStream keeps everything like a transaction register, KTable keeps only the latest like a balance board, and the right choice depends on whether your data is events or state.
---