Definition
- KStream-KStream join combines two event streams within a time window; both sides must be co-partitioned and a JoinWindows parameter defines how far apart records can be in time.
- KStream-KTable join enriches each stream record with the latest value from a table; the table is a lookup and no window is needed.
- KStream-GlobalKTable join is like KStream-KTable but the GlobalKTable is replicated to every instance, removing the co-partitioning requirement.
- KTable-KTable join produces a new table where each key's value is the result of joining both tables; an update on either side emits a new joined value.
- Co-partitioning is the requirement that both sides of a join have the same number of partitions and the same partitioning key, so matching records land on the same task.
- JoinWindows defines the time range for KStream-KStream joins: a record from stream A matches records from stream B that arrived within the specified before and after duration.
Think of National Bank of India's loan approval process. A KStream-KStream join is like matching a loan application (from stream A) with the credit score report (from stream B) that arrives within 24 hours of the application. Both documents flow in independently, and the join window is the 24-hour matching period. A KStream-KTable join is like the clerk checking Priya's current account balance (KTable) every time a new withdrawal request (KStream) arrives. The balance is always available; no window is needed. A KStream-GlobalKTable join is like a clerk checking the branch directory (GlobalKTable) for any branch code, even if the directory is maintained at head office and copied to every branch. A KTable-KTable join is like merging the customer profile table with the KYC status table so that every customer row has both their profile and verification status.
Understanding the Concepts
What are joins in Kafka Streams?
Joins combine records from two sources by key. When a record from the left side has the same key as a record from the right side, the join function produces a combined result. Kafka Streams supports joins between KStream-KStream, KStream-KTable, KStream-GlobalKTable, and KTable-KTable.
Each join type has different semantics: whether a time window is required, whether co-partitioning is needed, and what happens when one side has no matching record.
package com.bank.streams;
public class JoinTypesOverview {
public static void main(String[] args) {
System.out.println("Join types in Kafka Streams:");
System.out.println(" 1. KStream-KStream : windowed, co-partitioned");
System.out.println(" 2. KStream-KTable : lookup, co-partitioned");
System.out.println(" 3. KStream-GlobalKTable : lookup, no co-partition needed");
System.out.println(" 4. KTable-KTable : table merge, co-partitioned");
System.out.println();
System.out.println("Each supports: inner, left, outer join variants");
}
}
Expected output:
Join types in Kafka Streams:
1. KStream-KStream : windowed, co-partitioned
2. KStream-KTable : lookup, co-partitioned
3. KStream-GlobalKTable : lookup, no co-partition needed
4. KTable-KTable : table merge, co-partitioned
Each supports: inner, left, outer join variants
How does KStream-KStream join work?
KStream-KStream join combines two event streams within a time window. Both sides are unbounded, so you must provide a JoinWindows parameter that defines how far apart in time two records can be and still match.
When a record arrives on one side, Kafka Streams looks for matching records (same key) on the other side that fall within the window. Both sides are buffered in state stores for the duration of the window.
At National Bank of India, Rajesh matches debit events from one account with corresponding credit events on another account. A debit on SAV2001 should match a credit on CUR3001 within 10 minutes.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.kstream.StreamJoined;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.time.Duration;
public class StreamStreamJoinExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Debit events keyed by transfer ID
KStream<String, String> debits = builder.stream("debit-events");
// Credit events keyed by transfer ID
KStream<String, String> credits = builder.stream("credit-events");
// Join: match debit and credit within 10-minute window
KStream<String, String> matched = debits.join(
credits,
(debitValue, creditValue) ->
"MATCHED|" + debitValue + "|" + creditValue,
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10)),
StreamJoined.with(Serdes.String(), Serdes.String(), Serdes.String())
);
matched.to("matched-transfers");
Topology topology = builder.build();
System.out.println("KStream-KStream join:");
System.out.println(" Left: debit-events");
System.out.println(" Right: credit-events");
System.out.println(" Window: 10 minutes");
System.out.println(" Output: matched-transfers");
}
}
Expected output:
KStream-KStream join:
Left: debit-events
Right: credit-events
Window: 10 minutes
Output: matched-transfers
Example:
Debit: TXN001 -> SAV2001|DEBIT|50000|to CUR3001 (at 09:00)
Credit: TXN001 -> CUR3001|CREDIT|50000|from SAV2001 (at 09:03)
Matched: TXN001 -> MATCHED|SAV2001|DEBIT|50000|...|CUR3001|CREDIT|50000|...
Within 10-minute window, so join succeeds.
How does KStream-KTable join work?
KStream-KTable join enriches each stream record with the latest value from a table. When a record arrives on the KStream side, Kafka Streams looks up the matching key in the KTable's state store and produces a joined result. No time window is needed because the KTable always holds the current state.
This is the most common join in banking: enrich each transaction with the customer's current profile.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.Topology;
public class StreamTableJoinExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Transactions keyed by account ID
KStream<String, String> transactions = builder.stream("bank-transactions");
// Customer profiles keyed by account ID
KTable<String, String> profiles = builder.table("customer-profiles");
// Join: enrich each transaction with customer name
KStream<String, String> enriched = transactions.join(
profiles,
(txnValue, profileValue) ->
txnValue + "|CUSTOMER:" + profileValue
);
enriched.to("enriched-transactions");
Topology topology = builder.build();
System.out.println("KStream-KTable join:");
System.out.println(" Stream: bank-transactions (key=accountId)");
System.out.println(" Table: customer-profiles (key=accountId)");
System.out.println(" No window needed. Table lookup is instant.");
}
}
Expected output:
KStream-KTable join:
Stream: bank-transactions (key=accountId)
Table: customer-profiles (key=accountId)
No window needed. Table lookup is instant.
Example:
Transaction: SAV2001|DEPOSIT|50000|salary credit
Profile: Priya Sharma|Mumbai|KYC_VERIFIED
Enriched: SAV2001|DEPOSIT|50000|salary credit|CUSTOMER:Priya Sharma|Mumbai|KYC_VERIFIED
How does KStream-GlobalKTable join work?
KStream-GlobalKTable join is similar to KStream-KTable, but the GlobalKTable is replicated in full to every application instance. This removes the co-partitioning requirement. You can join on any key, not just the stream's key.
The tradeoff is that GlobalKTable uses more memory because every instance holds the entire dataset. Use it for small reference data like branch directories, currency codes, or country lists.
At National Bank of India, Rajesh joins transactions with the branch directory. The transactions are keyed by account ID, but the branch directory is keyed by branch code. A GlobalKTable lets him use a custom key extractor to map account ID to branch code.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.GlobalKTable;
import org.apache.kafka.streams.Topology;
public class StreamGlobalTableJoinExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Transactions keyed by account ID
KStream<String, String> transactions = builder.stream("bank-transactions");
// Branch directory: keyed by branch code (SAV, CUR, FD)
GlobalKTable<String, String> branchDirectory =
builder.globalTable("branch-directory");
// Join using a key mapper: extract branch code from account ID
KStream<String, String> enriched = transactions.join(
branchDirectory,
(accountId, txnValue) -> accountId.substring(0, 3), // key mapper
(txnValue, branchValue) -> txnValue + "|BRANCH:" + branchValue
);
enriched.to("enriched-with-branch");
Topology topology = builder.build();
System.out.println("KStream-GlobalKTable join:");
System.out.println(" Stream: bank-transactions (key=accountId)");
System.out.println(" GlobalKTable: branch-directory (key=branchCode)");
System.out.println(" Key mapper: accountId -> accountId.substring(0,3)");
System.out.println(" Co-partitioning: NOT required");
}
}
Expected output:
KStream-GlobalKTable join:
Stream: bank-transactions (key=accountId)
GlobalKTable: branch-directory (key=branchCode)
Key mapper: accountId -> accountId.substring(0,3)
Co-partitioning: NOT required
Example:
Transaction: SAV2001|DEPOSIT|50000|salary credit
Branch dir: SAV -> Savings Branch|Mumbai|Open 9-5
Enriched: SAV2001|DEPOSIT|50000|salary credit|BRANCH:Savings Branch|Mumbai|Open 9-5
How does KTable-KTable join work?
KTable-KTable join produces a new KTable where each key's value is the combined result of both tables. When either side is updated, the join re-evaluates for that key and emits a new result. Both sides must be co-partitioned.
At National Bank of India, Rajesh joins the customer profile table with the KYC status table. Whenever Priya's profile or KYC status changes, the joined result is updated.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.Topology;
public class TableTableJoinExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Customer profiles (key = customerId)
KTable<String, String> profiles = builder.table("customer-profiles");
// KYC status (key = customerId)
KTable<String, String> kyc = builder.table("kyc-status");
// Join: combine profile with KYC status
KTable<String, String> combined = profiles.join(
kyc,
(profileValue, kycValue) -> profileValue + "|KYC:" + kycValue
);
combined.toStream().to("customer-complete");
Topology topology = builder.build();
System.out.println("KTable-KTable join:");
System.out.println(" Left: customer-profiles (key=customerId)");
System.out.println(" Right: kyc-status (key=customerId)");
System.out.println(" Output: customer-complete");
System.out.println(" Update on either side emits new joined result.");
}
}
Expected output:
KTable-KTable join:
Left: customer-profiles (key=customerId)
Right: kyc-status (key=customerId)
Output: customer-complete
Update on either side emits new joined result.
Example:
Profile: SAV2001 -> Priya Sharma|Mumbai
KYC: SAV2001 -> VERIFIED|2024-01-15
Combined: SAV2001 -> Priya Sharma|Mumbai|KYC:VERIFIED|2024-01-15
When KYC is updated:
KYC: SAV2001 -> EXPIRED|2024-12-15
Combined: SAV2001 -> Priya Sharma|Mumbai|KYC:EXPIRED|2024-12-15
How do all join types compare?
| Feature | KStream-KStream | KStream-KTable | KStream-GlobalKTable | KTable-KTable |
|---|---|---|---|---|
| Window required | Yes (JoinWindows) | No | No | No |
| Co-partitioning | Yes | Yes | No | Yes |
| State stores | 2 (one per side) | 1 (table side) | 1 (global, full copy) | 2 (one per side) |
| Trigger | Record on either side | Record on stream side | Record on stream side | Record on either side |
| Result type | KStream | KStream | KStream | KTable |
| Inner join | Both sides must match | Stream matches table | Stream matches global | Both sides must match |
| Left join | Left always emits | Stream always emits | Stream always emits | Left always emits |
| Outer join | Both sides always emit | Not supported | Not supported | Both sides always emit |
| Banking use case | Match debit-credit | Enrich txn with profile | Enrich txn with branch | Merge profile + KYC |
package com.bank.streams;
public class JoinComparisonSummary {
public static void main(String[] args) {
System.out.println("Join decision guide:");
System.out.println();
System.out.println("Two event streams, time-bounded match?");
System.out.println(" -> KStream-KStream join with JoinWindows");
System.out.println();
System.out.println("Enrich events with current state (same key)?");
System.out.println(" -> KStream-KTable join");
System.out.println();
System.out.println("Enrich events with reference data (different key)?");
System.out.println(" -> KStream-GlobalKTable join");
System.out.println();
System.out.println("Merge two state tables?");
System.out.println(" -> KTable-KTable join");
}
}
Expected output:
Join decision guide:
Two event streams, time-bounded match?
-> KStream-KStream join with JoinWindows
Enrich events with current state (same key)?
-> KStream-KTable join
Enrich events with reference data (different key)?
-> KStream-GlobalKTable join
Merge two state tables?
-> KTable-KTable join
What are the co-partitioning rules?
Co-partitioning means both sides of a join must have the same number of partitions and use the same partitioning strategy (default: murmur2 hash on the key). This ensures that records with the same key end up on the same task and can be joined locally.
If the two topics have different partition counts or different keys, the join will produce incorrect or missing results. Kafka Streams does not validate this at startup; incorrect co-partitioning is a silent bug.
package com.bank.streams;
public class CoPartitioningRules {
public static void main(String[] args) {
System.out.println("Co-partitioning rules for joins:");
System.out.println();
System.out.println("Rule 1: Same number of partitions");
System.out.println(" bank-transactions: 6 partitions");
System.out.println(" customer-profiles: 6 partitions -> OK");
System.out.println(" customer-profiles: 3 partitions -> WRONG");
System.out.println();
System.out.println("Rule 2: Same partitioning key");
System.out.println(" bank-transactions key: accountId");
System.out.println(" customer-profiles key: accountId -> OK");
System.out.println(" customer-profiles key: name -> WRONG");
System.out.println();
System.out.println("Rule 3: Same partitioner (default: murmur2)");
System.out.println(" If one topic was produced with a custom partitioner,");
System.out.println(" records may not land on the expected partition.");
System.out.println();
System.out.println("Exception: GlobalKTable does not need co-partitioning");
System.out.println("because the entire table is replicated to every instance.");
}
}
Expected output:
Co-partitioning rules for joins:
Rule 1: Same number of partitions
bank-transactions: 6 partitions
customer-profiles: 6 partitions -> OK
customer-profiles: 3 partitions -> WRONG
Rule 2: Same partitioning key
bank-transactions key: accountId
customer-profiles key: accountId -> OK
customer-profiles key: name -> WRONG
Rule 3: Same partitioner (default: murmur2)
If one topic was produced with a custom partitioner,
records may not land on the expected partition.
Exception: GlobalKTable does not need co-partitioning
because the entire table is replicated to every instance.
What are join windows for KStream-KStream joins?
JoinWindows defines the time range for matching records in a KStream-KStream join. You specify how far before and after the arrival of a record on one side to look for a match on the other side.
The window is symmetric by default: JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10)) means a record on the left matches records on the right that arrived within 10 minutes before or after it. You can make it asymmetric with .before() and .after().
Both sides are buffered in state stores for the duration of the window. Larger windows use more memory and disk.
package com.bank.streams;
import org.apache.kafka.streams.kstream.JoinWindows;
import java.time.Duration;
public class JoinWindowExamples {
public static void main(String[] args) {
// Symmetric: 10 minutes before and after
JoinWindows symmetric =
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10));
System.out.println("Symmetric window: +/- 10 minutes");
System.out.println(" Debit at 09:05 matches credit from 08:55 to 09:15");
System.out.println();
// Asymmetric: 5 minutes before, 30 minutes after
JoinWindows asymmetric =
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(30))
.before(Duration.ofMinutes(5))
.after(Duration.ofMinutes(30));
System.out.println("Asymmetric window: 5 min before, 30 min after");
System.out.println(" Debit at 09:05 matches credit from 09:00 to 09:35");
System.out.println();
System.out.println("State store usage:");
System.out.println(" Both sides buffered for window duration");
System.out.println(" Larger window = more memory and disk");
}
}
Expected output:
Symmetric window: +/- 10 minutes
Debit at 09:05 matches credit from 08:55 to 09:15
Asymmetric window: 5 min before, 30 min after
Debit at 09:05 matches credit from 09:00 to 09:35
State store usage:
Both sides buffered for window duration
Larger window = more memory and disk
What is a complete banking project using joins?
- This project is a Transaction Enrichment Pipeline for National Bank of India. It performs three joins:
- KStream-KTable: enrich transactions with customer profiles
- KStream-GlobalKTable: enrich with branch directory
- KStream-KStream: match debit-credit pairs within a 10-minute window
Project structure
txn-enrichment-pipeline/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── TxnEnrichmentApp.java
├── TxnEnrichmentTopology.java
└── TxnEnrichmentTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>txn-enrichment-pipelinesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
TxnEnrichmentTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.GlobalKTable;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.kstream.StreamJoined;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
import java.time.Duration;
public class TxnEnrichmentTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// Source streams and tables
KStream<String, String> transactions = builder.stream("bank-transactions");
KTable<String, String> profiles = builder.table("customer-profiles");
GlobalKTable<String, String> branchDir = builder.globalTable("branch-directory");
// Join 1: KStream-KTable (enrich with customer profile)
KStream<String, String> withProfile = transactions.leftJoin(
profiles,
(txnValue, profileValue) -> {
String profile = profileValue != null ? profileValue : "UNKNOWN";
return txnValue + "|PROFILE:" + profile;
}
);
// Join 2: KStream-GlobalKTable (enrich with branch info)
KStream<String, String> withBranch = withProfile.join(
branchDir,
(accountId, txnValue) -> accountId.substring(0, 3),
(txnValue, branchValue) -> txnValue + "|BRANCH:" + branchValue
);
withBranch.to("enriched-transactions");
// Join 3: KStream-KStream (match debit-credit pairs)
KStream<String, String> debits = builder.stream("debit-events");
KStream<String, String> credits = builder.stream("credit-events");
KStream<String, String> matched = debits.join(
credits,
(debitValue, creditValue) ->
"MATCHED|" + debitValue + "|" + creditValue,
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10)),
StreamJoined.with(Serdes.String(), Serdes.String(), Serdes.String())
);
matched.to("matched-transfers");
return builder.build();
}
}
TxnEnrichmentApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class TxnEnrichmentApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"txn-enrichment-pipeline");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
TxnEnrichmentTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
}));
try {
streams.start();
System.out.println("Transaction enrichment pipeline started");
System.out.println(" Join 1: transactions + customer profiles (KStream-KTable)");
System.out.println(" Join 2: + branch directory (KStream-GlobalKTable)");
System.out.println(" Join 3: debit + credit matching (KStream-KStream, 10 min)");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
Expected output (on startup):
Transaction enrichment pipeline started
Join 1: transactions + customer profiles (KStream-KTable)
Join 2: + branch directory (KStream-GlobalKTable)
Join 3: debit + credit matching (KStream-KStream, 10 min)
TxnEnrichmentTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Instant;
import java.util.Properties;
public class TxnEnrichmentTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(TxnEnrichmentTopology.build(), props)) {
// Set up input topics
TestInputTopic<String, String> profiles = driver.createInputTopic(
"customer-profiles", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> branchDir = driver.createInputTopic(
"branch-directory", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> transactions = driver.createInputTopic(
"bank-transactions", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> debits = driver.createInputTopic(
"debit-events", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> credits = driver.createInputTopic(
"credit-events", new StringSerializer(), new StringSerializer());
// Output topics
TestOutputTopic<String, String> enriched = driver.createOutputTopic(
"enriched-transactions", new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> matched = driver.createOutputTopic(
"matched-transfers", new StringDeserializer(), new StringDeserializer());
// Populate reference data first
profiles.pipeInput("SAV2001", "Priya Sharma|Mumbai");
branchDir.pipeInput("SAV", "Savings Branch|Mumbai|Open 9-5");
// Send a transaction
transactions.pipeInput("SAV2001", "SAV2001|DEPOSIT|50000|salary credit");
System.out.println("=== Enriched Transactions ===");
System.out.println("Count: " + enriched.getQueueSize());
if (!enriched.isEmpty()) {
System.out.println("Record: " + enriched.readValue());
}
// Send debit and credit pair
Instant now = Instant.now();
debits.pipeInput("TXN001", "SAV2001|DEBIT|50000|to CUR3001", now);
credits.pipeInput("TXN001", "CUR3001|CREDIT|50000|from SAV2001",
now.plusSeconds(120));
System.out.println();
System.out.println("=== Matched Transfers ===");
System.out.println("Count: " + matched.getQueueSize());
if (!matched.isEmpty()) {
System.out.println("Record: " + matched.readValue());
}
}
}
}
Expected output:
=== Enriched Transactions ===
Count: 1
Record: SAV2001|DEPOSIT|50000|salary credit|PROFILE:Priya Sharma|Mumbai|BRANCH:Savings Branch|Mumbai|Open 9-5
=== Matched Transfers ===
Count: 1
Record: MATCHED|SAV2001|DEBIT|50000|to CUR3001|CUR3001|CREDIT|50000|from SAV2001
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics (all with same partition count for co-partitioning)
kafka-topics.sh --create --topic bank-transactions --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic customer-profiles --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic branch-directory --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic debit-events --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic credit-events --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic enriched-transactions --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic matched-transfers --partitions 6 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build and run
cd txn-enrichment-pipeline
mvn clean package
java -cp target/txn-enrichment-pipeline-1.0.0.jar com.bank.streams.TxnEnrichmentApp
What are common mistakes with joins?
The most common mistake is not co-partitioning the topics before joining.
Wrong: joining topics with different partition counts
package com.bank.streams;
public class WrongPartitionMismatch {
public static void main(String[] args) {
System.out.println("WRONG: Joining topics with different partition counts");
System.out.println();
System.out.println(" bank-transactions: 6 partitions");
System.out.println(" customer-profiles: 3 partitions");
System.out.println();
System.out.println("Problem:");
System.out.println(" SAV2001 hashes to partition 2 in the 6-partition topic");
System.out.println(" SAV2001 hashes to partition 1 in the 3-partition topic");
System.out.println(" Different tasks own these partitions.");
System.out.println(" The join silently produces no result for SAV2001.");
System.out.println();
System.out.println("Kafka Streams does NOT validate partition counts at startup.");
System.out.println("This is a silent bug.");
}
}
Expected output:
WRONG: Joining topics with different partition counts
bank-transactions: 6 partitions
customer-profiles: 3 partitions
Problem:
SAV2001 hashes to partition 2 in the 6-partition topic
SAV2001 hashes to partition 1 in the 3-partition topic
Different tasks own these partitions.
The join silently produces no result for SAV2001.
Kafka Streams does NOT validate partition counts at startup.
This is a silent bug.
Correct: ensuring co-partitioning
package com.bank.streams;
public class CorrectCoPartitioning {
public static void main(String[] args) {
System.out.println("CORRECT: Ensure co-partitioning before joining");
System.out.println();
System.out.println("Checklist:");
System.out.println(" 1. Create both topics with the same partition count");
System.out.println(" kafka-topics.sh --create --topic bank-transactions --partitions 6");
System.out.println(" kafka-topics.sh --create --topic customer-profiles --partitions 6");
System.out.println();
System.out.println(" 2. Produce with the same key on both topics");
System.out.println(" key = accountId on both sides");
System.out.println();
System.out.println(" 3. Use the same partitioner (default murmur2 on both)");
System.out.println();
System.out.println(" 4. If keys differ, use selectKey() + through() to repartition");
System.out.println(" before joining. Or use GlobalKTable to avoid the issue.");
}
}
Expected output:
CORRECT: Ensure co-partitioning before joining
Checklist:
1. Create both topics with the same partition count
kafka-topics.sh --create --topic bank-transactions --partitions 6
kafka-topics.sh --create --topic customer-profiles --partitions 6
2. Produce with the same key on both topics
key = accountId on both sides
3. Use the same partitioner (default murmur2 on both)
4. If keys differ, use selectKey() + through() to repartition
before joining. Or use GlobalKTable to avoid the issue.
What should a senior or principal engineer know?
At the senior level, you should understand the operational implications of each join type and how to optimize joins in production.
- State store cost of joins: KStream-KStream joins create two state stores (one per side) plus two changelog topics. A 10-minute window on a stream with 100K records/second buffers up to 60M records per side. Size your disk accordingly.
- GlobalKTable memory: The entire GlobalKTable is loaded into memory on every instance. If the branch directory has 10,000 entries, this is fine. If the customer profiles table has 50 million entries, use a KTable with co-partitioning instead.
- Left join null handling: In a left join, the right side value can be null. Always add null checks in the join function. A NullPointerException in the join function kills the stream thread.
- Foreign key joins (Kafka 2.4+): KTable-KTable joins can use a foreign key instead of the primary key. This is useful for joining accounts with customers when the account table has a customerId column that does not match the key.
- Grace period: KStream-KStream join windows have a grace period. Records arriving after the window closes plus the grace period are dropped. Set the grace period based on your expected maximum lateness.
- Join ordering for KStream-KTable: The KTable must receive its records before the KStream records arrive for the join to produce results. If the KStream record arrives first and the KTable has no entry yet, an inner join produces no output. Use a left join if you want the stream record to pass through even without a table match.
package com.bank.streams;
public class SeniorJoinKnowledge {
public static void main(String[] args) {
System.out.println("Senior checklist for Kafka Streams joins:");
System.out.println(" 1. Verify co-partitioning: same partition count + same key");
System.out.println(" 2. Size disks for KStream-KStream window buffering");
System.out.println(" 3. Use GlobalKTable only for small reference data");
System.out.println(" 4. Add null checks in left join functions");
System.out.println(" 5. Populate KTable before KStream for inner joins");
System.out.println(" 6. Set grace period for late-arriving records");
System.out.println(" 7. Monitor: join-rate, window-buffer-size, changelog-lag");
}
}
Expected output:
Senior checklist for Kafka Streams joins:
1. Verify co-partitioning: same partition count + same key
2. Size disks for KStream-KStream window buffering
3. Use GlobalKTable only for small reference data
4. Add null checks in left join functions
5. Populate KTable before KStream for inner joins
6. Set grace period for late-arriving records
7. Monitor: join-rate, window-buffer-size, changelog-lag
What are common interview follow-up questions?
- Can you do a three-way join in Kafka Streams? (Yes. Chain two joins: first join stream A with table B, then join the result with table C. Each join is a separate step in the topology.)
- What happens if the GlobalKTable topic has millions of records? (Every instance loads the entire table into memory. With 50 million records at 500 bytes each, that is 25 GB per instance. Use a regular KTable with co-partitioning instead.)
- Can you join a KStream with a KStream without a window? (No. Both sides are unbounded. Without a window, the join would need to buffer all records forever, which is not feasible.)
- What is the difference between inner join and left join for KStream-KTable? (Inner join: produces output only when the KTable has a matching key. Left join: always produces output for the KStream record, with null for the table side if no match exists.)
- Does KStream-KTable join support outer join? (No. Only inner and left joins are supported. The KTable side cannot trigger the join; only KStream records trigger lookups.)
- How do you handle different keys for KStream-KTable join? (Use selectKey() to rekey the KStream to match the KTable's key. This triggers repartitioning. Alternatively, use a GlobalKTable with a key mapper to avoid repartitioning.)
Visual Overview
KStream-KStream Join (Windowed)
==================================
┌──────────────┐ ┌──────────────┐
│ debit-events │ │ credit-events│
└──────┬───────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Window Store │ │ Window Store │
│ (10 min buf) │ │ (10 min buf) │
└──────┬───────┘ └──────┬───────┘
│ match by key │
└────────┬────────────┘
│
▼
┌────────────────┐
│ matched- │
│ transfers │
└────────────────┘
KStream-KTable Join (Lookup)
===============================
┌──────────────┐ ┌──────────────┐
│ bank- │ │ customer- │
│ transactions │ │ profiles │
└──────┬───────┘ └──────┬───────┘
│ │
│ ┌─────┴──────┐
│ │ State Store│
│ │ (latest │
│ │ per key) │
│ └─────┬──────┘
│ lookup by key │
└────────┬───────────┘
│
▼
┌────────────────┐
│ enriched- │
│ transactions │
└────────────────┘
KStream-GlobalKTable Join
============================
┌──────────────┐ ┌──────────────────┐
│ bank- │ │ branch-directory │
│ transactions │ │ (GlobalKTable) │
└──────┬───────┘ │ Full copy on │
│ │ every instance │
│ └──────┬───────────┘
│ key mapper │
│ accountId->branch │
└────────┬───────────┘
│
▼
┌────────────────┐
│ enriched-with- │
│ branch │
└────────────────┘
No co-partitioning needed!
Key Takeaway
Joins are how Kafka Streams combines data from multiple sources into a single enriched result. The four join types serve different use cases, and choosing the right one depends on whether you are joining events with events, events with state, or state with state.
KStream-KStream joins match records from two event streams within a time window. At National Bank of India, this is how Rajesh matches debit events with corresponding credit events. Both sides are buffered in state stores for the duration of the window, so larger windows use more memory. KStream-KTable joins are the most common pattern in banking: each transaction is enriched with the customer's current profile by looking up the account ID in the KTable's state store. No window is needed because the KTable always holds the latest state.
KStream-GlobalKTable joins remove the co-partitioning requirement because the GlobalKTable is replicated in full to every instance. This is useful for small reference data like branch directories or currency codes, but the tradeoff is memory: every instance holds the entire dataset. KTable-KTable joins produce a new table where each key's value is the combined result of both input tables, and updates on either side trigger a new output.
The most critical operational concern is co-partitioning. All joins except KStream-GlobalKTable require both sides to have the same number of partitions and the same partitioning key. Kafka Streams does not validate this at startup, so a partition count mismatch is a silent bug that produces missing join results. Always verify partition counts and key structure before deploying a join topology.
At the senior level, size your disks for KStream-KStream window buffering, use GlobalKTable only for small reference data, handle null values in left joins, and ensure the KTable is populated before KStream records arrive for inner joins.
Remember one sentence: joins combine records by key, KStream-KStream needs a window, KStream-KTable needs co-partitioning, KStream-GlobalKTable needs neither, and co-partitioning is the most common source of silent join bugs.
---