Definition
- KStream models an unbounded sequence of events where each record is a standalone fact; three records with key SAV2001 mean three events.
- KTable models a continuously updated table where each key has exactly one current value; three records with key SAV2001 mean the value was updated twice, and only the last one is the current state.
- Null handling differs: KStream passes null values through as regular records; KTable interprets a null value as a delete (tombstone) and removes the key from its state store.
- State store: KTable always maintains a local key-value store backed by a changelog topic; KStream does not maintain state unless you call groupByKey and aggregate.
- Joining: a KStream-KTable join enriches each streaming event with the latest lookup value from the table, similar to a SQL stream-table join.
At National Bank of India, Rajesh's savings account has two views. The transaction register (KStream) records every deposit and withdrawal: 50,000 credited on Monday, 30,000 credited on Tuesday, 10,000 debited on Wednesday. All three entries exist side by side. The passbook summary (KTable) shows the current balance: 70,000. If a new deposit arrives, the transaction register adds a new line. The passbook updates the balance to the new total. The register is the full history. The passbook is the current state.
Understanding the Concepts
How does KStream process records differently from KTable?
The fundamental difference is in how each abstraction treats records with the same key.
KStream appends every record. If three records arrive for key SAV2001, the KStream processes all three independently. The output has three records.
KTable upserts. If three records arrive for key SAV2001, the KTable state store holds only the latest value. Downstream consumers see three change events, but the state store has one entry.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.streams.state.ValueAndTimestamp;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class KStreamVsKTableSideBySlide {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
// === KStream test ===
StreamsBuilder streamBuilder = new StreamsBuilder();
KStream<String, String> stream = streamBuilder.stream("transactions");
stream.to("stream-output");
try (TopologyTestDriver streamDriver =
new TopologyTestDriver(streamBuilder.build(), props)) {
TestInputTopic<String, String> sIn = streamDriver.createInputTopic(
"transactions", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> sOut = streamDriver.createOutputTopic(
"stream-output", new StringDeserializer(), new StringDeserializer());
sIn.pipeInput("SAV2001", "DEPOSIT|50000");
sIn.pipeInput("SAV2001", "DEPOSIT|30000");
sIn.pipeInput("SAV2001", "WITHDRAWAL|10000");
System.out.println("=== KStream output ===");
System.out.println("Records: " + sOut.getQueueSize());
while (!sOut.isEmpty()) {
System.out.println(" " + sOut.readValue());
}
}
// === KTable test ===
StreamsBuilder tableBuilder = new StreamsBuilder();
KTable<String, String> table = tableBuilder.table(
"balances", Materialized.as("balance-store"));
table.toStream().to("table-output");
try (TopologyTestDriver tableDriver =
new TopologyTestDriver(tableBuilder.build(), props)) {
TestInputTopic<String, String> tIn = tableDriver.createInputTopic(
"balances", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> tOut = tableDriver.createOutputTopic(
"table-output", new StringDeserializer(), new StringDeserializer());
tIn.pipeInput("SAV2001", "50000");
tIn.pipeInput("SAV2001", "80000");
tIn.pipeInput("SAV2001", "70000");
KeyValueStore<String, ValueAndTimestamp<String>> store =
tableDriver.getTimestampedKeyValueStore("balance-store");
System.out.println();
System.out.println("=== KTable output ===");
System.out.println("Records emitted: " + tOut.getQueueSize());
while (!tOut.isEmpty()) {
System.out.println(" " + tOut.readValue());
}
System.out.println("State store (current value): " + store.get("SAV2001").value());
}
}
}
Expected output:
=== KStream output ===
Records: 3
DEPOSIT|50000
DEPOSIT|30000
WITHDRAWAL|10000
=== KTable output ===
Records emitted: 3
50000
80000
70000
State store (current value): 70000
KStream output has 3 records (all events kept). KTable also emits 3 change events, but the state store holds only the latest value: 70000.
What is the complete comparison between KStream and KTable?
| Feature | KStream | KTable |
|---|---|---|
| Semantics | Event stream (INSERT log) | Changelog (UPSERT table) |
| Record meaning | Each record is a new event | Each record updates the key's value |
| Same key, 3 records | 3 events in output | 3 change events, but state has 1 entry |
| Null value | Regular record (passed through) | Tombstone (deletes the key) |
| State store | None by default | Always present (RocksDB) |
| Created by | builder.stream() | builder.table() or groupByKey().aggregate() |
| Source topic type | Any (delete or compact) | Compacted recommended |
| Memory usage | Low (no state) | Proportional to unique key count |
| Startup time | Fast (no state to restore) | Slow if state store is large |
| Use case | Transactions, payments, logins | Balances, profiles, statuses |
| SQL analogy | INSERT INTO events ... | UPSERT INTO table ... |
package com.bank.streams;
public class ComparisonSummary {
public static void main(String[] args) {
System.out.println("Decision guide:");
System.out.println();
System.out.println(" Is every event important? -> KStream");
System.out.println(" Do you need current state? -> KTable");
System.out.println(" Is this a transaction log? -> KStream");
System.out.println(" Is this a balance/status/profile? -> KTable");
System.out.println(" Are duplicate keys updates? -> KTable");
System.out.println(" Are duplicate keys new events? -> KStream");
}
}
Expected output:
Decision guide:
Is every event important? -> KStream
Do you need current state? -> KTable
Is this a transaction log? -> KStream
Is this a balance/status/profile? -> KTable
Are duplicate keys updates? -> KTable
Are duplicate keys new events? -> KStream
How do null values behave differently in KStream and KTable?
In KStream, a null value is just another record. It flows through the pipeline like any other event. In KTable, a null value is a tombstone: it deletes the key from the state store.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.streams.state.ValueAndTimestamp;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class NullValueDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
StreamsBuilder builder = new StreamsBuilder();
KTable<String, String> accounts = builder.table(
"accounts", Materialized.as("account-store"));
accounts.toStream().to("account-changes");
try (TopologyTestDriver driver =
new TopologyTestDriver(builder.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"accounts", new StringSerializer(), new StringSerializer());
// Insert account
input.pipeInput("SAV2001", "Rajesh|50000|ACTIVE");
// Close account (null = tombstone)
input.pipeInput("SAV2001", (String) null);
KeyValueStore<String, ValueAndTimestamp<String>> store =
driver.getTimestampedKeyValueStore("account-store");
ValueAndTimestamp<String> result = store.get("SAV2001");
System.out.println("KTable state after tombstone:");
System.out.println(" SAV2001: " + (result != null ? result.value() : "DELETED"));
System.out.println();
System.out.println("In KStream: null value is a regular record (passed through)");
System.out.println("In KTable: null value = tombstone (key deleted from store)");
}
}
}
Expected output:
KTable state after tombstone:
SAV2001: DELETED
In KStream: null value is a regular record (passed through)
In KTable: null value = tombstone (key deleted from store)
What is a complete banking project demonstrating KStream and KTable together?
This project builds a transaction enrichment service. Transactions arrive as a KStream. Customer profiles are maintained as a KTable. A KStream-KTable join enriches each transaction with the customer's name and account type.
Project structure
transaction-enrichment-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── EnrichmentApp.java
│ └── EnrichmentTopology.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── EnrichmentTest.java
EnrichmentTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
public class EnrichmentTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// KStream: every transaction is an event
KStream<String, String> transactions = builder.stream("transactions");
// KTable: latest customer profile per account ID
KTable<String, String> customers = builder.table("customer-profiles");
// Join: enrich each transaction with customer name
KStream<String, String> enriched = transactions.join(
customers,
(transaction, profile) -> {
// transaction: DEPOSIT|50000|Mumbai
// profile: Rajesh Singh|SAVINGS|Premium
String customerName = profile.split("\\|")[0];
return transaction + "|CUSTOMER:" + customerName;
}
);
enriched.to("enriched-transactions");
return builder.build();
}
}
EnrichmentTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class EnrichmentTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(EnrichmentTopology.build(), props)) {
TestInputTopic<String, String> profiles = driver.createInputTopic(
"customer-profiles",
new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> txns = driver.createInputTopic(
"transactions",
new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"enriched-transactions",
new StringDeserializer(), new StringDeserializer());
// Load customer profiles into KTable first
profiles.pipeInput("SAV2001", "Rajesh Singh|SAVINGS|Premium");
profiles.pipeInput("CUR3001", "Priya Sharma|CURRENT|Business");
// Send transactions (joined with profile from KTable)
txns.pipeInput("SAV2001", "DEPOSIT|50000|Mumbai");
txns.pipeInput("CUR3001", "NEFT|75000|Delhi");
// Update profile (KTable replaces old value)
profiles.pipeInput("SAV2001", "Rajesh Singh|SAVINGS|Platinum");
// New transaction uses updated profile
txns.pipeInput("SAV2001", "WITHDRAWAL|10000|Mumbai");
System.out.println("Enriched transactions:");
while (!output.isEmpty()) {
var record = output.readKeyValue();
System.out.println(" " + record.key + ": " + record.value);
}
System.out.println();
System.out.println("Test passed:");
System.out.println(" KStream events enriched with KTable customer profiles");
System.out.println(" Profile update reflected in subsequent transactions");
}
}
}
Expected output:
Enriched transactions:
SAV2001: DEPOSIT|50000|Mumbai|CUSTOMER:Rajesh Singh
CUR3001: NEFT|75000|Delhi|CUSTOMER:Priya Sharma
SAV2001: WITHDRAWAL|10000|Mumbai|CUSTOMER:Rajesh Singh
Test passed:
KStream events enriched with KTable customer profiles
Profile update reflected in subsequent transactions
How to run
# Step 1: Create topics (customer-profiles should be compacted)
kafka-topics.sh --create --topic customer-profiles --partitions 4 --replication-factor 1 --config cleanup.policy=compact --bootstrap-server localhost:9092
kafka-topics.sh --create --topic transactions --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic enriched-transactions --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
# Step 2: Build and test
cd transaction-enrichment-service
mvn clean package
java -cp target/transaction-enrichment-service-1.0.0.jar com.bank.streams.EnrichmentTest
# Step 3: Run
java -cp target/transaction-enrichment-service-1.0.0.jar com.bank.streams.EnrichmentApp
What are common mistakes when choosing between KStream and KTable?
Wrong: reading a balance topic as KStream
package com.bank.streams;
public class WrongBalanceAsKStream {
public static void main(String[] args) {
System.out.println("WRONG: builder.stream('account-balances')");
System.out.println();
System.out.println(" What the developer wanted:");
System.out.println(" Current balance per account for joins");
System.out.println();
System.out.println(" What they got:");
System.out.println(" Every historical balance update as separate events");
System.out.println(" Join matches on stale and current values");
System.out.println(" No state store for lookups");
System.out.println();
System.out.println(" Fix: use builder.table('account-balances')");
System.out.println(" KTable keeps only the current balance per key");
System.out.println(" Joins always use the latest value");
}
}
Expected output:
WRONG: builder.stream('account-balances')
What the developer wanted:
Current balance per account for joins
What they got:
Every historical balance update as separate events
Join matches on stale and current values
No state store for lookups
Fix: use builder.table('account-balances')
KTable keeps only the current balance per key
Joins always use the latest value
Correct: match the abstraction to the data semantics
package com.bank.streams;
public class CorrectAbstractionChoice {
public static void main(String[] args) {
System.out.println("CORRECT: Choose based on data semantics");
System.out.println();
System.out.println(" Transactions (events): builder.stream()");
System.out.println(" Account balances (state): builder.table()");
System.out.println(" Customer profiles (state): builder.table()");
System.out.println(" Payment events (events): builder.stream()");
System.out.println(" Order status (state): builder.table()");
System.out.println(" Audit log (events): builder.stream()");
System.out.println(" Product catalog (state): builder.table()");
}
}
Expected output:
CORRECT: Choose based on data semantics
Transactions (events): builder.stream()
Account balances (state): builder.table()
Customer profiles (state): builder.table()
Payment events (events): builder.stream()
Order status (state): builder.table()
Audit log (events): builder.stream()
Product catalog (state): builder.table()
What should a senior or principal engineer know?
At the senior level, you should be able to decide between KStream and KTable by looking at the data semantics: events go to KStream, state goes to KTable. You should know that KStream-KTable joins use the KTable as a lookup.
At the principal level, consider:
- Co-partitioning requirement: KStream-KTable joins require both topics to have the same number of partitions and the same partitioning strategy. If they differ, the join produces incorrect results silently.
- KTable source topic must be compacted: without compaction, state restoration reads the full history, which is slow for large topics.
- KTable caching: rapid updates to the same key are deduplicated in the cache before being flushed to the state store and emitted downstream. This reduces I/O but adds latency. Control with
cache.max.bytes.buffering. - Foreign-key joins (Kafka 2.4+): KTable-KTable join on a non-primary key. Useful for joining accounts with a branch table where the join key is branch ID, not account ID.
- Versioned state stores (Kafka 3.5+): store historical versions of KTable values, enabling point-in-time queries.
package com.bank.streams;
public class PrincipalKStreamKTableKnowledge {
public static void main(String[] args) {
System.out.println("Principal-level KStream vs KTable knowledge:");
System.out.println();
System.out.println("1. Co-partitioning for joins:");
System.out.println(" transactions (4 partitions) + profiles (4 partitions) = OK");
System.out.println(" transactions (4 partitions) + profiles (6 partitions) = WRONG");
System.out.println(" Both must have same partition count + same key + same partitioner");
System.out.println();
System.out.println("2. Compacted source for KTable:");
System.out.println(" cleanup.policy=compact on the source topic");
System.out.println(" Reduces state restoration time after crash");
System.out.println();
System.out.println("3. Foreign-key join (Kafka 2.4+):");
System.out.println(" accounts.join(branches, account -> account.getBranchId(), ...)");
System.out.println(" Join key is branch ID, not account ID");
System.out.println();
System.out.println("4. Monitoring:");
System.out.println(" KTable state store size: state-store-metrics");
System.out.println(" KStream throughput: process-rate, records-lag-max");
}
}
Expected output:
Principal-level KStream vs KTable knowledge:
1. Co-partitioning for joins:
transactions (4 partitions) + profiles (4 partitions) = OK
transactions (4 partitions) + profiles (6 partitions) = WRONG
Both must have same partition count + same key + same partitioner
2. Compacted source for KTable:
cleanup.policy=compact on the source topic
Reduces state restoration time after crash
3. Foreign-key join (Kafka 2.4+):
accounts.join(branches, account -> account.getBranchId(), ...)
Join key is branch ID, not account ID
4. Monitoring:
KTable state store size: state-store-metrics
KStream throughput: process-rate, records-lag-max
How does a KStream-KTable join work internally?
When you join a KStream with a KTable, Kafka Streams looks up the KStream record's key in the KTable's local state store. If the key exists, the ValueJoiner combines both values. If the key does not exist, the behavior depends on the join type: inner join drops the record, left join passes the KStream value with a null KTable value.
The KTable must be co-partitioned with the KStream: same number of partitions, same key type, same partitioning strategy.
package com.bank.streams;
public class JoinSemantics {
public static void main(String[] args) {
System.out.println("KStream-KTable join semantics:");
System.out.println();
System.out.println(" Inner join (stream.join(table, joiner)):");
System.out.println(" KStream key found in KTable -> joiner called, result emitted");
System.out.println(" KStream key NOT in KTable -> record dropped");
System.out.println();
System.out.println(" Left join (stream.leftJoin(table, joiner)):");
System.out.println(" KStream key found in KTable -> joiner called, result emitted");
System.out.println(" KStream key NOT in KTable -> joiner called with null table value");
System.out.println();
System.out.println(" Requirement: KStream and KTable must be co-partitioned");
System.out.println(" Same partition count + same key + same partitioner");
}
}
Expected output:
KStream-KTable join semantics:
Inner join (stream.join(table, joiner)):
KStream key found in KTable -> joiner called, result emitted
KStream key NOT in KTable -> record dropped
Left join (stream.leftJoin(table, joiner)):
KStream key found in KTable -> joiner called, result emitted
KStream key NOT in KTable -> joiner called with null table value
Requirement: KStream and KTable must be co-partitioned
Same partition count + same key + same partitioner
What are common interview follow-up questions?
- Can you convert a KStream to a KTable? (Yes. Call groupByKey() then aggregate(), reduce(), or count(). The result is a KTable holding the accumulated state.)
- Can you convert a KTable to a KStream? (Yes. Call toStream(). Each state change in the KTable becomes an event in the KStream.)
- What is the performance cost of KTable vs KStream? (KTable maintains a state store, which uses disk (RocksDB) and memory. KStream is stateless by default and uses no disk. KTable also has a changelog topic, doubling write I/O.)
- What happens in a KStream-KTable join if the KTable key does not exist? (The KStream record is dropped in an inner join. In a left join, the KStream record passes through with a null value from the KTable side.)
- Can a KTable have more entries than the source topic? (No. KTable entries are bounded by the number of unique keys in the source topic. Compaction keeps only the latest value per key.)
- What is a GlobalKTable and when would you use it instead of KTable? (GlobalKTable replicates the entire topic to every instance. Use it for small lookup tables like currency codes or branch lists where co-partitioning is not possible.)
Visual Overview
KStream vs KTable: Same Input, Different View
================================================
Input: 3 records for key SAV2001
SAV2001 -> 50000
SAV2001 -> 80000
SAV2001 -> 70000
┌────────────────────────────┐
│ KStream │
│ │
│ SAV2001 -> 50000 (event 1)│
│ SAV2001 -> 80000 (event 2)│
│ SAV2001 -> 70000 (event 3)│
│ │
│ All 3 records kept │
│ No state store │
└────────────────────────────┘
┌────────────────────────────┐
│ KTable │
│ │
│ State Store: │
│ ┌─────────┬───────┐ │
│ │ SAV2001 │ 70000 │ │
│ └─────────┴───────┘ │
│ │
│ Only latest value kept │
│ 50000 and 80000 replaced │
└────────────────────────────┘
KStream-KTable Join
====================
KStream (transactions) KTable (profiles)
┌──────────────────┐ ┌──────────────────┐
│ SAV2001 DEPOSIT │ │ SAV2001 Rajesh │
│ CUR3001 NEFT │ │ CUR3001 Priya │
└────────┬─────────┘ └────────┬─────────┘
│ │
└────────┬───────────────┘
│ join on key
▼
┌──────────────────────────────────┐
│ SAV2001 DEPOSIT|CUSTOMER:Rajesh │
│ CUR3001 NEFT|CUSTOMER:Priya │
└──────────────────────────────────┘
Key Takeaway
KStream and KTable are the two core abstractions in Kafka Streams, and choosing between them depends on the data semantics. KStream is for events: every record is an independent fact, and duplicate keys mean multiple events. KTable is for state: each key has one current value, and new records replace old ones.
The practical difference shows clearly in a banking context. Transactions are a KStream because every deposit and withdrawal matters. Account balances are a KTable because you only need the current balance. When you need to enrich transactions with customer data, you join the KStream with a KTable: each transaction picks up the latest customer profile.
In our National Bank of India enrichment project, transactions flowed as a KStream, customer profiles lived in a KTable, and a join produced enriched transactions with customer names. When a profile was updated, subsequent transactions automatically picked up the new data.
At the principal level, ensure co-partitioning for joins (same partition count and key), use compacted source topics for KTable, and consider foreign-key joins when the join key is not the primary key.
Remember one sentence: KStream is the transaction log where every event is a fact, KTable is the balance sheet where each key has one current value.
---