Definition
- Inner join emits a result record only when a record from the left stream and a record from the right stream share the same key and arrive within the join window.
- Left join emits a result for every left-side record; if no matching right-side record exists within the window, the right value is null.
- Outer join emits a result for every record on either side; if the other side has no match within the window, its value is null.
- JoinWindows defines the time window for KStream-KStream joins; records must arrive within this window to be considered a match.
- ValueJoiner is the function you provide to combine matching left and right values into a single output value.
Think of National Bank of India reconciling debits and credits at end of day. An inner join is like the reconciliation clerk who only marks a debit as matched when a corresponding credit slip exists. A left join is like the debit auditor who lists every debit and writes "unmatched" next to debits with no credit. An outer join is like the full audit report that lists every debit and every credit, marking "unmatched" on whichever side has no partner.
Understanding the Concepts
What is a join in Kafka Streams?
A join in Kafka Streams combines records from two streams or a stream and a table based on a common key. When two records share the same key and meet the join condition, Kafka Streams passes both values to a ValueJoiner function that produces the output record.
Kafka Streams supports three join patterns:
- KStream-KStream join: both sides are event streams, a time window is required.
- KStream-KTable join: the stream is enriched with the latest table value per key, no window needed.
- KTable-KTable join: both sides are changelog tables, result updates whenever either side changes.
package com.bank.streams;
public class JoinOverview {
public static void main(String[] args) {
System.out.println("Kafka Streams join types:");
System.out.println(" 1. KStream-KStream: both are event streams, window required");
System.out.println(" 2. KStream-KTable: stream enriched by table, no window");
System.out.println(" 3. KTable-KTable: both are tables, updates on either side");
System.out.println();
System.out.println("Each type supports: inner, left, outer");
System.out.println("All joins require co-partitioning (same key, same partition count)");
}
}
Expected output:
Kafka Streams join types:
1. KStream-KStream: both are event streams, window required
2. KStream-KTable: stream enriched by table, no window
3. KTable-KTable: both are tables, updates on either side
Each type supports: inner, left, outer
All joins require co-partitioning (same key, same partition count)
What is an inner join in Kafka Streams?
An inner join produces an output record only when both the left side and the right side have a matching record with the same key. If a debit arrives but no credit matches within the window, no output is produced. If a credit arrives but no debit matches, no output is produced.
This is the strictest join type. It guarantees that every output record has a non-null value from both sides.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class InnerJoinExample {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> debits = builder.stream("debits");
KStream<String, String> credits = builder.stream("credits");
// Inner join: output only when both debit and credit exist for same key
KStream<String, String> matched = debits.join(
credits,
(debitValue, creditValue) -> "MATCHED|" + debitValue + "|" + creditValue,
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
);
matched.to("reconciled-transactions");
return builder.build();
}
public static void main(String[] args) {
Topology topology = build();
System.out.println("Inner Join Topology:");
System.out.println(topology.describe());
}
}
Expected output:
Inner Join Topology:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [debits])
--> KSTREAM-WINDOWED-0000000002
Source: KSTREAM-SOURCE-0000000001 (topics: [credits])
--> KSTREAM-WINDOWED-0000000003
Processor: KSTREAM-WINDOWED-0000000002 (stores: [KSTREAM-JOINTHIS-...])
--> KSTREAM-MERGE-0000000004
Processor: KSTREAM-WINDOWED-0000000003 (stores: [KSTREAM-JOINOTHER-...])
--> KSTREAM-MERGE-0000000004
Processor: KSTREAM-MERGE-0000000004
--> KSTREAM-SINK-0000000005
Sink: KSTREAM-SINK-0000000005 (topic: reconciled-transactions)
What is a left join in Kafka Streams?
A left join produces an output record for every left-side record. If a matching right-side record exists within the window, both values are passed to the joiner. If no right-side match exists, the right value is null.
This is useful when you want to process every record from the primary stream and optionally enrich it. For example, every debit transaction is processed, and if a matching credit exists, it is attached.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class LeftJoinExample {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> debits = builder.stream("debits");
KStream<String, String> credits = builder.stream("credits");
// Left join: every debit produces output; credit may be null
KStream<String, String> enriched = debits.leftJoin(
credits,
(debitValue, creditValue) -> {
if (creditValue == null) {
return "UNMATCHED|" + debitValue + "|NO_CREDIT";
}
return "MATCHED|" + debitValue + "|" + creditValue;
},
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
);
enriched.to("debit-audit");
return builder.build();
}
public static void main(String[] args) {
System.out.println("Left Join behavior:");
System.out.println(" Debit arrives, credit exists -> MATCHED|debit|credit");
System.out.println(" Debit arrives, no credit -> UNMATCHED|debit|NO_CREDIT");
System.out.println(" Credit arrives, no debit -> no output (right side ignored)");
}
}
Expected output:
Left Join behavior:
Debit arrives, credit exists -> MATCHED|debit|credit
Debit arrives, no credit -> UNMATCHED|debit|NO_CREDIT
Credit arrives, no debit -> no output (right side ignored)
What is an outer join in Kafka Streams?
An outer join produces an output record whenever a record arrives on either side. If the other side has a matching record within the window, both values are present. If not, the missing side is null.
This gives you a complete picture of all activity on both streams. It is useful for full audit trails where you need to see every debit and every credit, whether matched or not.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class OuterJoinExample {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> debits = builder.stream("debits");
KStream<String, String> credits = builder.stream("credits");
// Outer join: output for every record on both sides
KStream<String, String> fullAudit = debits.outerJoin(
credits,
(debitValue, creditValue) -> {
if (debitValue == null) {
return "CREDIT_ONLY|NO_DEBIT|" + creditValue;
}
if (creditValue == null) {
return "DEBIT_ONLY|" + debitValue + "|NO_CREDIT";
}
return "MATCHED|" + debitValue + "|" + creditValue;
},
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
);
fullAudit.to("full-audit");
return builder.build();
}
public static void main(String[] args) {
System.out.println("Outer Join behavior:");
System.out.println(" Debit + credit matched -> MATCHED|debit|credit");
System.out.println(" Debit only, no credit -> DEBIT_ONLY|debit|NO_CREDIT");
System.out.println(" Credit only, no debit -> CREDIT_ONLY|NO_DEBIT|credit");
}
}
Expected output:
Outer Join behavior:
Debit + credit matched -> MATCHED|debit|credit
Debit only, no credit -> DEBIT_ONLY|debit|NO_CREDIT
Credit only, no debit -> CREDIT_ONLY|NO_DEBIT|credit
How do inner, left, and outer joins differ?
The three join types differ in when they produce output and how they handle missing records.
| Feature | Inner Join | Left Join | Outer Join |
|---|---|---|---|
| Output when both sides match | Yes | Yes | Yes |
| Output when only left arrives | No | Yes (right = null) | Yes (right = null) |
| Output when only right arrives | No | No | Yes (left = null) |
| Null handling required | No nulls | Right may be null | Either may be null |
| Use case | Strict matching | Enrichment | Full audit trail |
| Method | join() | leftJoin() | outerJoin() |
| Guarantees | Both values present | Left always present | At least one present |
package com.bank.streams;
public class JoinComparisonDemo {
public static void main(String[] args) {
// Simulating three scenarios
String debitTxn = "SAV2001|DEBIT|50000";
String creditTxn = "SAV2001|CREDIT|50000";
System.out.println("Scenario 1: Both debit and credit arrive");
System.out.println(" Inner: MATCHED|" + debitTxn + "|" + creditTxn);
System.out.println(" Left: MATCHED|" + debitTxn + "|" + creditTxn);
System.out.println(" Outer: MATCHED|" + debitTxn + "|" + creditTxn);
System.out.println();
System.out.println("Scenario 2: Only debit arrives (no credit within window)");
System.out.println(" Inner: <no output>");
System.out.println(" Left: UNMATCHED|" + debitTxn + "|NO_CREDIT");
System.out.println(" Outer: DEBIT_ONLY|" + debitTxn + "|NO_CREDIT");
System.out.println();
System.out.println("Scenario 3: Only credit arrives (no debit within window)");
System.out.println(" Inner: <no output>");
System.out.println(" Left: <no output>");
System.out.println(" Outer: CREDIT_ONLY|NO_DEBIT|" + creditTxn);
}
}
Expected output:
Scenario 1: Both debit and credit arrive
Inner: MATCHED|SAV2001|DEBIT|50000|SAV2001|CREDIT|50000
Left: MATCHED|SAV2001|DEBIT|50000|SAV2001|CREDIT|50000
Outer: MATCHED|SAV2001|DEBIT|50000|SAV2001|CREDIT|50000
Scenario 2: Only debit arrives (no credit within window)
Inner: <no output>
Left: UNMATCHED|SAV2001|DEBIT|50000|NO_CREDIT
Outer: DEBIT_ONLY|SAV2001|DEBIT|50000|NO_CREDIT
Scenario 3: Only credit arrives (no debit within window)
Inner: <no output>
Left: <no output>
Outer: CREDIT_ONLY|NO_DEBIT|SAV2001|CREDIT|50000
How do you handle null values in join results?
In a left join, the right-side value can be null. In an outer join, either side can be null. You must check for null inside the ValueJoiner to avoid NullPointerException.
A safe pattern is to always check both sides and produce a status field that indicates whether the record is matched, debit-only, or credit-only.
package com.bank.streams;
public class NullHandlingInJoins {
static String safeJoin(String debit, String credit) {
if (debit != null && credit != null) {
return "RECONCILED|" + debit + "|" + credit;
}
if (debit != null) {
return "PENDING_CREDIT|" + debit;
}
if (credit != null) {
return "PENDING_DEBIT|" + credit;
}
return "ERROR|both_null";
}
public static void main(String[] args) {
System.out.println("Test 1 (both present): " + safeJoin("DEBIT|50000", "CREDIT|50000"));
System.out.println("Test 2 (credit null): " + safeJoin("DEBIT|50000", null));
System.out.println("Test 3 (debit null): " + safeJoin(null, "CREDIT|50000"));
System.out.println("Test 4 (both null): " + safeJoin(null, null));
}
}
Expected output:
Test 1 (both present): RECONCILED|DEBIT|50000|CREDIT|50000
Test 2 (credit null): PENDING_CREDIT|DEBIT|50000
Test 3 (debit null): PENDING_DEBIT|CREDIT|50000
Test 4 (both null): ERROR|both_null
What is a complete banking project using joins for reconciliation?
This project reads debit transactions from one topic and credit transactions from another topic. It uses an outer join to match debits with credits by transaction reference ID within a 10-minute window. The output topic contains matched pairs, unmatched debits, and unmatched credits for the reconciliation team at National Bank of India.
Project structure
reconciliation-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── ReconciliationApp.java
│ ├── ReconciliationTopology.java
│ └── TransactionRecord.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── ReconciliationTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>reconciliation-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
TransactionRecord.java
package com.bank.streams;
import java.math.BigDecimal;
public class TransactionRecord {
private String refId;
private String accountId;
private String type;
private BigDecimal amount;
public TransactionRecord() {}
public TransactionRecord(String refId, String accountId,
String type, BigDecimal amount) {
this.refId = refId;
this.accountId = accountId;
this.type = type;
this.amount = amount;
}
public String getRefId() { return refId; }
public String getAccountId() { return accountId; }
public String getType() { return type; }
public BigDecimal getAmount() { return amount; }
@Override
public String toString() {
return refId + "|" + accountId + "|" + type + "|" + amount;
}
public static TransactionRecord fromString(String value) {
String[] parts = value.split("\\|");
return new TransactionRecord(
parts[0], parts[1], parts[2], new BigDecimal(parts[3]));
}
}
ReconciliationTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class ReconciliationTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> debits = builder.stream("debits");
KStream<String, String> credits = builder.stream("credits");
KStream<String, String> reconciled = debits.outerJoin(
credits,
(debitValue, creditValue) -> {
if (debitValue != null && creditValue != null) {
return "RECONCILED|" + debitValue + "|" + creditValue;
}
if (debitValue != null) {
return "PENDING_CREDIT|" + debitValue;
}
if (creditValue != null) {
return "PENDING_DEBIT|" + creditValue;
}
return "ERROR|BOTH_NULL";
},
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10))
);
reconciled.to("reconciliation-results");
return builder.build();
}
}
ReconciliationApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class ReconciliationApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "reconciliation-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
ReconciliationTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
System.out.println("Reconciliation service shut down");
}));
try {
streams.start();
System.out.println("Reconciliation service started");
System.out.println(" Debit topic: debits");
System.out.println(" Credit topic: credits");
System.out.println(" Output topic: reconciliation-results");
System.out.println(" Join window: 10 minutes");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
ReconciliationTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Instant;
import java.util.Properties;
public class ReconciliationTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(ReconciliationTopology.build(), props)) {
TestInputTopic<String, String> debitTopic = driver.createInputTopic(
"debits", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> creditTopic = driver.createInputTopic(
"credits", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> outputTopic = driver.createOutputTopic(
"reconciliation-results",
new StringDeserializer(), new StringDeserializer());
Instant now = Instant.now();
// Test 1: Matched debit and credit
debitTopic.pipeInput("TXN001",
"TXN001|SAV2001|DEBIT|50000", now);
creditTopic.pipeInput("TXN001",
"TXN001|SAV2001|CREDIT|50000", now.plusSeconds(60));
// Test 2: Debit with no matching credit
debitTopic.pipeInput("TXN002",
"TXN002|CUR3001|DEBIT|75000", now.plusSeconds(120));
// Test 3: Credit with no matching debit
creditTopic.pipeInput("TXN003",
"TXN003|SAV2002|CREDIT|30000", now.plusSeconds(180));
System.out.println("Output records:");
while (!outputTopic.isEmpty()) {
System.out.println(" " + outputTopic.readValue());
}
}
}
}
Expected output:
Output records:
PENDING_CREDIT|TXN001|SAV2001|DEBIT|50000
RECONCILED|TXN001|SAV2001|DEBIT|50000|TXN001|SAV2001|CREDIT|50000
PENDING_CREDIT|TXN002|CUR3001|DEBIT|75000
PENDING_DEBIT|TXN003|SAV2002|CREDIT|30000
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics with same partition count (co-partitioning)
kafka-topics.sh --create --topic debits --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic credits --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic reconciliation-results --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build
cd reconciliation-service
mvn clean package
# Step 4: Run
java -cp target/reconciliation-service-1.0.0.jar com.bank.streams.ReconciliationApp
# Step 5: Produce debits (key:value)
kafka-console-producer.sh --topic debits --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# TXN001:TXN001|SAV2001|DEBIT|50000
# TXN002:TXN002|CUR3001|DEBIT|75000
# Step 6: Produce credits
kafka-console-producer.sh --topic credits --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# TXN001:TXN001|SAV2001|CREDIT|50000
# Step 7: Consume results
kafka-console-consumer.sh --topic reconciliation-results --from-beginning --bootstrap-server localhost:9092
What are common mistakes when using joins in Kafka Streams?
The most common mistake is not handling null values in left and outer joins.
Wrong: no null check in outer join ValueJoiner
package com.bank.streams;
public class WrongJoinNoNullCheck {
public static void main(String[] args) {
// Simulating what happens without null check
String debitValue = "TXN001|SAV2001|DEBIT|50000";
String creditValue = null; // No matching credit
try {
// WRONG: calling methods on potentially null creditValue
String result = "MATCHED|" + debitValue + "|" + creditValue.toUpperCase();
System.out.println(result);
} catch (NullPointerException e) {
System.out.println("NullPointerException thrown!");
System.out.println("Cause: creditValue is null in left/outer join");
System.out.println("Fix: always check for null before using join values");
}
}
}
Expected output:
NullPointerException thrown!
Cause: creditValue is null in left/outer join
Fix: always check for null before using join values
Correct: null-safe ValueJoiner
package com.bank.streams;
public class CorrectJoinWithNullCheck {
static String reconcile(String debit, String credit) {
if (debit != null && credit != null) {
return "RECONCILED|" + debit + "|" + credit;
}
if (debit != null) {
return "PENDING_CREDIT|" + debit;
}
if (credit != null) {
return "PENDING_DEBIT|" + credit;
}
return "ERROR|BOTH_NULL";
}
public static void main(String[] args) {
System.out.println("Test matched: " + reconcile("DEBIT|50000", "CREDIT|50000"));
System.out.println("Test debit only: " + reconcile("DEBIT|50000", null));
System.out.println("Test credit only: " + reconcile(null, "CREDIT|50000"));
System.out.println();
System.out.println("Lesson: always handle null in left join and outer join ValueJoiners");
}
}
Expected output:
Test matched: RECONCILED|DEBIT|50000|CREDIT|50000
Test debit only: PENDING_CREDIT|DEBIT|50000
Test credit only: PENDING_DEBIT|CREDIT|50000
Lesson: always handle null in left join and outer join ValueJoiners
What about testing joins with TopologyTestDriver?
TopologyTestDriver lets you test all three join types without a running Kafka broker. You create input topics for both sides of the join, pipe records with specific timestamps, and verify the output.
The key to testing joins is controlling timestamps. You pipe records at specific instants and verify that records inside the window produce joined output and records outside the window do not.
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Instant;
import java.util.Properties;
public class JoinTestWithTimestamps {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(InnerJoinExample.build(), props)) {
TestInputTopic<String, String> debitTopic = driver.createInputTopic(
"debits", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> creditTopic = driver.createInputTopic(
"credits", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> outputTopic = driver.createOutputTopic(
"reconciled-transactions",
new StringDeserializer(), new StringDeserializer());
Instant baseTime = Instant.parse("2024-01-15T10:00:00Z");
// Within window (3 minutes apart, window is 5 minutes)
debitTopic.pipeInput("TXN100", "DEBIT|SAV2001|50000", baseTime);
creditTopic.pipeInput("TXN100", "CREDIT|SAV2001|50000",
baseTime.plusSeconds(180));
// Outside window (15 minutes apart, window is 5 minutes)
debitTopic.pipeInput("TXN200", "DEBIT|CUR3001|75000", baseTime);
creditTopic.pipeInput("TXN200", "CREDIT|CUR3001|75000",
baseTime.plusSeconds(900));
System.out.println("Inner join results:");
System.out.println(" Records in output: " + outputTopic.getQueueSize());
while (!outputTopic.isEmpty()) {
System.out.println(" " + outputTopic.readValue());
}
System.out.println(" TXN100: matched (within 5-min window)");
System.out.println(" TXN200: no output (outside 5-min window)");
}
}
}
Expected output:
Inner join results:
Records in output: 1
MATCHED|DEBIT|SAV2001|50000|CREDIT|SAV2001|50000
TXN100: matched (within 5-min window)
TXN200: no output (outside 5-min window)
What should a senior or principal engineer know?
At the senior level, you need to understand the performance implications of each join type and how to choose between them.
- Inner join creates the least state because it only stores records that have a chance of matching. It is the most efficient choice when you need strict matching.
- Outer join creates the most state because every record from both sides must be stored for the window duration. In high-throughput systems, this can cause large RocksDB state stores.
- Window size directly affects memory usage. A 10-minute window stores all records from both sides for 10 minutes. A 1-hour window stores 6 times more data.
- Spurious results: in KStream-KStream joins, the first match emits immediately, but if another matching record arrives within the window, another result is emitted. This means you may see intermediate results before the final matched pair. Use the
SuppressedAPI if you need only the final result. - Repartitioning cost: if the join key is different from the current key, Kafka Streams must repartition one or both streams, which creates internal topics and adds latency.
package com.bank.streams;
public class SeniorJoinKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal join checklist:");
System.out.println(" 1. Inner join = least state, outer join = most state");
System.out.println(" 2. Window size directly affects RocksDB memory");
System.out.println(" 3. KStream-KStream joins produce spurious intermediate results");
System.out.println(" 4. Use Suppressed API for final-result-only semantics");
System.out.println(" 5. Repartitioning for different join keys adds internal topics");
System.out.println(" 6. Co-partitioning is mandatory: same partition count, same key");
System.out.println(" 7. Monitor join state store size via JMX metrics");
System.out.println(" 8. Consider KStream-KTable join to avoid windowing entirely");
}
}
Expected output:
Senior/Principal join checklist:
1. Inner join = least state, outer join = most state
2. Window size directly affects RocksDB memory
3. KStream-KStream joins produce spurious intermediate results
4. Use Suppressed API for final-result-only semantics
5. Repartitioning for different join keys adds internal topics
6. Co-partitioning is mandatory: same partition count, same key
7. Monitor join state store size via JMX metrics
8. Consider KStream-KTable join to avoid windowing entirely
What are common interview follow-up questions?
- Can you join more than two streams? (Not directly. You chain two joins: join stream A with stream B, then join the result with stream C.)
- What happens if the key types are different? (The join fails at runtime. Both sides must use the same key serde.)
- Can you do a KStream-KTable left join? (Yes. Every stream record produces output. If no table entry exists for the key, the table side is null.)
- What is a GlobalKTable join? (A GlobalKTable is replicated to every instance. It does not require co-partitioning, but it uses more memory because every instance holds the full table.)
- How do you join on a non-key field? (You must repartition the stream using selectKey() to change the key to the join field before calling join().)
Visual Overview
Inner Join (KStream-KStream)
============================
┌──────────┐ ┌──────────┐
│ Debits │ │ Credits │
│ (stream) │ │ (stream) │
└────┬─────┘ └────┬─────┘
│ │
└───────┬────────┘
│ join()
▼
┌───────────────────┐
│ JoinWindows: 10m │
│ Both must match │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Output: RECONCILED│
│ (only if both │
│ sides present) │
└───────────────────┘
Left Join
=========
┌──────────┐ ┌──────────┐
│ Debits │ │ Credits │
│ (LEFT) │ │ (RIGHT) │
└────┬─────┘ └────┬─────┘
│ │
└───────┬────────┘
│ leftJoin()
▼
┌───────────────────┐
│ Every debit emits │
│ Credit = null OK │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ RECONCILED or │
│ PENDING_CREDIT │
└───────────────────┘
Outer Join
==========
┌──────────┐ ┌──────────┐
│ Debits │ │ Credits │
│ (stream) │ │ (stream) │
└────┬─────┘ └────┬─────┘
│ │
└───────┬────────┘
│ outerJoin()
▼
┌───────────────────┐
│ Every record from │
│ either side emits │
│ Other side = null │
│ is allowed │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ RECONCILED or │
│ PENDING_CREDIT or │
│ PENDING_DEBIT │
└───────────────────┘
Key Takeaway
Kafka Streams supports three join types for combining records from two streams or tables. An inner join is the strictest: it produces output only when both sides have a matching record within the join window. A left join always produces output for the left side, filling null for the right when no match exists. An outer join produces output for every record on either side, marking null for whichever side is missing.
In our reconciliation project at National Bank of India, we used an outer join to match debit and credit transactions by reference ID within a 10-minute window. Every debit and every credit appeared in the output, whether matched or not. The reconciliation team could see RECONCILED, PENDING_CREDIT, and PENDING_DEBIT statuses at a glance.
The most critical rule for joins is null handling. In a left join, the right value can be null. In an outer join, either value can be null. Forgetting to check for null is the number one cause of NullPointerException in join code. Always use a safe ValueJoiner that checks both sides before calling methods on them.
At the senior level, remember that join state grows with the window size: a larger window means more records stored in RocksDB. KStream-KStream joins can produce spurious intermediate results before the final match arrives. For final-result-only semantics, combine the join with the Suppressed API. Co-partitioning is mandatory for all joins: both input topics must have the same number of partitions and the same key.
Remember one sentence: inner join requires both sides, left join guarantees the left side, outer join guarantees both sides, and all three require co-partitioning and null-safe ValueJoiners.
---