Back to Kafka Streams
Q18Question 18 of 27

What is the difference between inner, left, and outer joins in Kafk

Answer: Inner join produces output only when both sides match, left join always produces output for the left side (with null if the right is missing), and outer join produces output for both sides (with null for whichever side is missing).

Key Concepts
  • Inner join emits a result only when both sides have a matching record within the join window.
  • Left join emits a result for every left-side record, filling null for the right side when no match exists.
  • Outer join emits a result whenever either side has a record, filling null for the missing side.
  • All three join types require co-partitioning: both input topics must have the same number of partitions and the same key.
  • KStream-KStream joins require a JoinWindows to define how far apart in time two records can still match.
  • KStream-KTable joins do not require a window because the table always holds the latest value per key.
  • Use inner join for strict matching like debit-credit reconciliation, left join for enrichment, and outer join for full audit trails.

Definition

Inner join produces output only when both sides match, left join always produces output for the left side (with null if the right is missing), and outer join produces output for both sides (with null for whichever side is missing).
  • Inner join emits a result record only when a record from the left stream and a record from the right stream share the same key and arrive within the join window.
  • Left join emits a result for every left-side record; if no matching right-side record exists within the window, the right value is null.
  • Outer join emits a result for every record on either side; if the other side has no match within the window, its value is null.
  • JoinWindows defines the time window for KStream-KStream joins; records must arrive within this window to be considered a match.
  • ValueJoiner is the function you provide to combine matching left and right values into a single output value.

Think of National Bank of India reconciling debits and credits at end of day. An inner join is like the reconciliation clerk who only marks a debit as matched when a corresponding credit slip exists. A left join is like the debit auditor who lists every debit and writes "unmatched" next to debits with no credit. An outer join is like the full audit report that lists every debit and every credit, marking "unmatched" on whichever side has no partner.

Understanding the Concepts

What is a join in Kafka Streams?

A join in Kafka Streams combines records from two streams or a stream and a table based on a common key. When two records share the same key and meet the join condition, Kafka Streams passes both values to a ValueJoiner function that produces the output record.

Kafka Streams supports three join patterns:

  • KStream-KStream join: both sides are event streams, a time window is required.
  • KStream-KTable join: the stream is enriched with the latest table value per key, no window needed.
  • KTable-KTable join: both sides are changelog tables, result updates whenever either side changes.
JAVA
package com.bank.streams;

public class JoinOverview {
    public static void main(String[] args) {
        System.out.println("Kafka Streams join types:");
        System.out.println("  1. KStream-KStream: both are event streams, window required");
        System.out.println("  2. KStream-KTable: stream enriched by table, no window");
        System.out.println("  3. KTable-KTable: both are tables, updates on either side");
        System.out.println();
        System.out.println("Each type supports: inner, left, outer");
        System.out.println("All joins require co-partitioning (same key, same partition count)");
    }
}
TEXT
Expected output:
Kafka Streams join types:
  1. KStream-KStream: both are event streams, window required
  2. KStream-KTable: stream enriched by table, no window
  3. KTable-KTable: both are tables, updates on either side

Each type supports: inner, left, outer
All joins require co-partitioning (same key, same partition count)

What is an inner join in Kafka Streams?

An inner join produces an output record only when both the left side and the right side have a matching record with the same key. If a debit arrives but no credit matches within the window, no output is produced. If a credit arrives but no debit matches, no output is produced.

This is the strictest join type. It guarantees that every output record has a non-null value from both sides.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class InnerJoinExample {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> debits = builder.stream("debits");
        KStream<String, String> credits = builder.stream("credits");

        // Inner join: output only when both debit and credit exist for same key
        KStream<String, String> matched = debits.join(
            credits,
            (debitValue, creditValue) -> "MATCHED|" + debitValue + "|" + creditValue,
            JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
        );

        matched.to("reconciled-transactions");
        return builder.build();
    }

    public static void main(String[] args) {
        Topology topology = build();
        System.out.println("Inner Join Topology:");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Inner Join Topology:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [debits])
      --> KSTREAM-WINDOWED-0000000002
    Source: KSTREAM-SOURCE-0000000001 (topics: [credits])
      --> KSTREAM-WINDOWED-0000000003
    Processor: KSTREAM-WINDOWED-0000000002 (stores: [KSTREAM-JOINTHIS-...])
      --> KSTREAM-MERGE-0000000004
    Processor: KSTREAM-WINDOWED-0000000003 (stores: [KSTREAM-JOINOTHER-...])
      --> KSTREAM-MERGE-0000000004
    Processor: KSTREAM-MERGE-0000000004
      --> KSTREAM-SINK-0000000005
    Sink: KSTREAM-SINK-0000000005 (topic: reconciled-transactions)

What is a left join in Kafka Streams?

A left join produces an output record for every left-side record. If a matching right-side record exists within the window, both values are passed to the joiner. If no right-side match exists, the right value is null.

This is useful when you want to process every record from the primary stream and optionally enrich it. For example, every debit transaction is processed, and if a matching credit exists, it is attached.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class LeftJoinExample {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> debits = builder.stream("debits");
        KStream<String, String> credits = builder.stream("credits");

        // Left join: every debit produces output; credit may be null
        KStream<String, String> enriched = debits.leftJoin(
            credits,
            (debitValue, creditValue) -> {
                if (creditValue == null) {
                    return "UNMATCHED|" + debitValue + "|NO_CREDIT";
                }
                return "MATCHED|" + debitValue + "|" + creditValue;
            },
            JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
        );

        enriched.to("debit-audit");
        return builder.build();
    }

    public static void main(String[] args) {
        System.out.println("Left Join behavior:");
        System.out.println("  Debit arrives, credit exists   -> MATCHED|debit|credit");
        System.out.println("  Debit arrives, no credit       -> UNMATCHED|debit|NO_CREDIT");
        System.out.println("  Credit arrives, no debit       -> no output (right side ignored)");
    }
}
TEXT
Expected output:
Left Join behavior:
  Debit arrives, credit exists   -> MATCHED|debit|credit
  Debit arrives, no credit       -> UNMATCHED|debit|NO_CREDIT
  Credit arrives, no debit       -> no output (right side ignored)

What is an outer join in Kafka Streams?

An outer join produces an output record whenever a record arrives on either side. If the other side has a matching record within the window, both values are present. If not, the missing side is null.

This gives you a complete picture of all activity on both streams. It is useful for full audit trails where you need to see every debit and every credit, whether matched or not.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class OuterJoinExample {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> debits = builder.stream("debits");
        KStream<String, String> credits = builder.stream("credits");

        // Outer join: output for every record on both sides
        KStream<String, String> fullAudit = debits.outerJoin(
            credits,
            (debitValue, creditValue) -> {
                if (debitValue == null) {
                    return "CREDIT_ONLY|NO_DEBIT|" + creditValue;
                }
                if (creditValue == null) {
                    return "DEBIT_ONLY|" + debitValue + "|NO_CREDIT";
                }
                return "MATCHED|" + debitValue + "|" + creditValue;
            },
            JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
        );

        fullAudit.to("full-audit");
        return builder.build();
    }

    public static void main(String[] args) {
        System.out.println("Outer Join behavior:");
        System.out.println("  Debit + credit matched  -> MATCHED|debit|credit");
        System.out.println("  Debit only, no credit   -> DEBIT_ONLY|debit|NO_CREDIT");
        System.out.println("  Credit only, no debit   -> CREDIT_ONLY|NO_DEBIT|credit");
    }
}
TEXT
Expected output:
Outer Join behavior:
  Debit + credit matched  -> MATCHED|debit|credit
  Debit only, no credit   -> DEBIT_ONLY|debit|NO_CREDIT
  Credit only, no debit   -> CREDIT_ONLY|NO_DEBIT|credit

How do inner, left, and outer joins differ?

The three join types differ in when they produce output and how they handle missing records.

FeatureInner JoinLeft JoinOuter Join
Output when both sides matchYesYesYes
Output when only left arrivesNoYes (right = null)Yes (right = null)
Output when only right arrivesNoNoYes (left = null)
Null handling requiredNo nullsRight may be nullEither may be null
Use caseStrict matchingEnrichmentFull audit trail
Methodjoin()leftJoin()outerJoin()
GuaranteesBoth values presentLeft always presentAt least one present
JAVA
package com.bank.streams;

public class JoinComparisonDemo {
    public static void main(String[] args) {
        // Simulating three scenarios
        String debitTxn = "SAV2001|DEBIT|50000";
        String creditTxn = "SAV2001|CREDIT|50000";

        System.out.println("Scenario 1: Both debit and credit arrive");
        System.out.println("  Inner:  MATCHED|" + debitTxn + "|" + creditTxn);
        System.out.println("  Left:   MATCHED|" + debitTxn + "|" + creditTxn);
        System.out.println("  Outer:  MATCHED|" + debitTxn + "|" + creditTxn);
        System.out.println();

        System.out.println("Scenario 2: Only debit arrives (no credit within window)");
        System.out.println("  Inner:  <no output>");
        System.out.println("  Left:   UNMATCHED|" + debitTxn + "|NO_CREDIT");
        System.out.println("  Outer:  DEBIT_ONLY|" + debitTxn + "|NO_CREDIT");
        System.out.println();

        System.out.println("Scenario 3: Only credit arrives (no debit within window)");
        System.out.println("  Inner:  <no output>");
        System.out.println("  Left:   <no output>");
        System.out.println("  Outer:  CREDIT_ONLY|NO_DEBIT|" + creditTxn);
    }
}
TEXT
Expected output:
Scenario 1: Both debit and credit arrive
  Inner:  MATCHED|SAV2001|DEBIT|50000|SAV2001|CREDIT|50000
  Left:   MATCHED|SAV2001|DEBIT|50000|SAV2001|CREDIT|50000
  Outer:  MATCHED|SAV2001|DEBIT|50000|SAV2001|CREDIT|50000

Scenario 2: Only debit arrives (no credit within window)
  Inner:  <no output>
  Left:   UNMATCHED|SAV2001|DEBIT|50000|NO_CREDIT
  Outer:  DEBIT_ONLY|SAV2001|DEBIT|50000|NO_CREDIT

Scenario 3: Only credit arrives (no debit within window)
  Inner:  <no output>
  Left:   <no output>
  Outer:  CREDIT_ONLY|NO_DEBIT|SAV2001|CREDIT|50000

How do you handle null values in join results?

In a left join, the right-side value can be null. In an outer join, either side can be null. You must check for null inside the ValueJoiner to avoid NullPointerException.

A safe pattern is to always check both sides and produce a status field that indicates whether the record is matched, debit-only, or credit-only.

JAVA
package com.bank.streams;

public class NullHandlingInJoins {

    static String safeJoin(String debit, String credit) {
        if (debit != null && credit != null) {
            return "RECONCILED|" + debit + "|" + credit;
        }
        if (debit != null) {
            return "PENDING_CREDIT|" + debit;
        }
        if (credit != null) {
            return "PENDING_DEBIT|" + credit;
        }
        return "ERROR|both_null";
    }

    public static void main(String[] args) {
        System.out.println("Test 1 (both present):  " + safeJoin("DEBIT|50000", "CREDIT|50000"));
        System.out.println("Test 2 (credit null):   " + safeJoin("DEBIT|50000", null));
        System.out.println("Test 3 (debit null):    " + safeJoin(null, "CREDIT|50000"));
        System.out.println("Test 4 (both null):     " + safeJoin(null, null));
    }
}
TEXT
Expected output:
Test 1 (both present):  RECONCILED|DEBIT|50000|CREDIT|50000
Test 2 (credit null):   PENDING_CREDIT|DEBIT|50000
Test 3 (debit null):    PENDING_DEBIT|CREDIT|50000
Test 4 (both null):     ERROR|both_null

What is a complete banking project using joins for reconciliation?

This project reads debit transactions from one topic and credit transactions from another topic. It uses an outer join to match debits with credits by transaction reference ID within a 10-minute window. The output topic contains matched pairs, unmatched debits, and unmatched credits for the reconciliation team at National Bank of India.

Project structure

DIAGRAM
reconciliation-service/
├── pom.xml
└── src/
    ├── main/
    │   └── java/
    │       └── com/
    │           └── bank/
    │               └── streams/
    │                   ├── ReconciliationApp.java
    │                   ├── ReconciliationTopology.java
    │                   └── TransactionRecord.java
    └── test/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        └── ReconciliationTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>reconciliation-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>
    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

TransactionRecord.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class TransactionRecord {
    private String refId;
    private String accountId;
    private String type;
    private BigDecimal amount;

    public TransactionRecord() {}

    public TransactionRecord(String refId, String accountId,
                             String type, BigDecimal amount) {
        this.refId = refId;
        this.accountId = accountId;
        this.type = type;
        this.amount = amount;
    }

    public String getRefId() { return refId; }
    public String getAccountId() { return accountId; }
    public String getType() { return type; }
    public BigDecimal getAmount() { return amount; }

    @Override
    public String toString() {
        return refId + "|" + accountId + "|" + type + "|" + amount;
    }

    public static TransactionRecord fromString(String value) {
        String[] parts = value.split("\\|");
        return new TransactionRecord(
            parts[0], parts[1], parts[2], new BigDecimal(parts[3]));
    }
}

ReconciliationTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class ReconciliationTopology {

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> debits = builder.stream("debits");
        KStream<String, String> credits = builder.stream("credits");

        KStream<String, String> reconciled = debits.outerJoin(
            credits,
            (debitValue, creditValue) -> {
                if (debitValue != null && creditValue != null) {
                    return "RECONCILED|" + debitValue + "|" + creditValue;
                }
                if (debitValue != null) {
                    return "PENDING_CREDIT|" + debitValue;
                }
                if (creditValue != null) {
                    return "PENDING_DEBIT|" + creditValue;
                }
                return "ERROR|BOTH_NULL";
            },
            JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10))
        );

        reconciled.to("reconciliation-results");
        return builder.build();
    }
}

ReconciliationApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class ReconciliationApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "reconciliation-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        KafkaStreams streams = new KafkaStreams(
            ReconciliationTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
            System.out.println("Reconciliation service shut down");
        }));

        try {
            streams.start();
            System.out.println("Reconciliation service started");
            System.out.println("  Debit topic:  debits");
            System.out.println("  Credit topic: credits");
            System.out.println("  Output topic: reconciliation-results");
            System.out.println("  Join window:  10 minutes");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

ReconciliationTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Instant;
import java.util.Properties;

public class ReconciliationTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(ReconciliationTopology.build(), props)) {

            TestInputTopic<String, String> debitTopic = driver.createInputTopic(
                "debits", new StringSerializer(), new StringSerializer());
            TestInputTopic<String, String> creditTopic = driver.createInputTopic(
                "credits", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> outputTopic = driver.createOutputTopic(
                "reconciliation-results",
                new StringDeserializer(), new StringDeserializer());

            Instant now = Instant.now();

            // Test 1: Matched debit and credit
            debitTopic.pipeInput("TXN001",
                "TXN001|SAV2001|DEBIT|50000", now);
            creditTopic.pipeInput("TXN001",
                "TXN001|SAV2001|CREDIT|50000", now.plusSeconds(60));

            // Test 2: Debit with no matching credit
            debitTopic.pipeInput("TXN002",
                "TXN002|CUR3001|DEBIT|75000", now.plusSeconds(120));

            // Test 3: Credit with no matching debit
            creditTopic.pipeInput("TXN003",
                "TXN003|SAV2002|CREDIT|30000", now.plusSeconds(180));

            System.out.println("Output records:");
            while (!outputTopic.isEmpty()) {
                System.out.println("  " + outputTopic.readValue());
            }
        }
    }
}
TEXT
Expected output:
Output records:
  PENDING_CREDIT|TXN001|SAV2001|DEBIT|50000
  RECONCILED|TXN001|SAV2001|DEBIT|50000|TXN001|SAV2001|CREDIT|50000
  PENDING_CREDIT|TXN002|CUR3001|DEBIT|75000
  PENDING_DEBIT|TXN003|SAV2002|CREDIT|30000

How to run the project

BASH
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper

# Step 2: Create topics with same partition count (co-partitioning)
kafka-topics.sh --create --topic debits --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic credits --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic reconciliation-results --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build
cd reconciliation-service
mvn clean package

# Step 4: Run
java -cp target/reconciliation-service-1.0.0.jar com.bank.streams.ReconciliationApp

# Step 5: Produce debits (key:value)
kafka-console-producer.sh --topic debits --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# TXN001:TXN001|SAV2001|DEBIT|50000
# TXN002:TXN002|CUR3001|DEBIT|75000

# Step 6: Produce credits
kafka-console-producer.sh --topic credits --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# TXN001:TXN001|SAV2001|CREDIT|50000

# Step 7: Consume results
kafka-console-consumer.sh --topic reconciliation-results --from-beginning --bootstrap-server localhost:9092

What are common mistakes when using joins in Kafka Streams?

The most common mistake is not handling null values in left and outer joins.

Wrong: no null check in outer join ValueJoiner

JAVA
package com.bank.streams;

public class WrongJoinNoNullCheck {
    public static void main(String[] args) {
        // Simulating what happens without null check
        String debitValue = "TXN001|SAV2001|DEBIT|50000";
        String creditValue = null; // No matching credit

        try {
            // WRONG: calling methods on potentially null creditValue
            String result = "MATCHED|" + debitValue + "|" + creditValue.toUpperCase();
            System.out.println(result);
        } catch (NullPointerException e) {
            System.out.println("NullPointerException thrown!");
            System.out.println("Cause: creditValue is null in left/outer join");
            System.out.println("Fix: always check for null before using join values");
        }
    }
}
TEXT
Expected output:
NullPointerException thrown!
Cause: creditValue is null in left/outer join
Fix: always check for null before using join values

Correct: null-safe ValueJoiner

JAVA
package com.bank.streams;

public class CorrectJoinWithNullCheck {
    static String reconcile(String debit, String credit) {
        if (debit != null && credit != null) {
            return "RECONCILED|" + debit + "|" + credit;
        }
        if (debit != null) {
            return "PENDING_CREDIT|" + debit;
        }
        if (credit != null) {
            return "PENDING_DEBIT|" + credit;
        }
        return "ERROR|BOTH_NULL";
    }

    public static void main(String[] args) {
        System.out.println("Test matched:     " + reconcile("DEBIT|50000", "CREDIT|50000"));
        System.out.println("Test debit only:  " + reconcile("DEBIT|50000", null));
        System.out.println("Test credit only: " + reconcile(null, "CREDIT|50000"));
        System.out.println();
        System.out.println("Lesson: always handle null in left join and outer join ValueJoiners");
    }
}
TEXT
Expected output:
Test matched:     RECONCILED|DEBIT|50000|CREDIT|50000
Test debit only:  PENDING_CREDIT|DEBIT|50000
Test credit only: PENDING_DEBIT|CREDIT|50000

Lesson: always handle null in left join and outer join ValueJoiners

What about testing joins with TopologyTestDriver?

TopologyTestDriver lets you test all three join types without a running Kafka broker. You create input topics for both sides of the join, pipe records with specific timestamps, and verify the output.

The key to testing joins is controlling timestamps. You pipe records at specific instants and verify that records inside the window produce joined output and records outside the window do not.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Instant;
import java.util.Properties;

public class JoinTestWithTimestamps {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(InnerJoinExample.build(), props)) {

            TestInputTopic<String, String> debitTopic = driver.createInputTopic(
                "debits", new StringSerializer(), new StringSerializer());
            TestInputTopic<String, String> creditTopic = driver.createInputTopic(
                "credits", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> outputTopic = driver.createOutputTopic(
                "reconciled-transactions",
                new StringDeserializer(), new StringDeserializer());

            Instant baseTime = Instant.parse("2024-01-15T10:00:00Z");

            // Within window (3 minutes apart, window is 5 minutes)
            debitTopic.pipeInput("TXN100", "DEBIT|SAV2001|50000", baseTime);
            creditTopic.pipeInput("TXN100", "CREDIT|SAV2001|50000",
                baseTime.plusSeconds(180));

            // Outside window (15 minutes apart, window is 5 minutes)
            debitTopic.pipeInput("TXN200", "DEBIT|CUR3001|75000", baseTime);
            creditTopic.pipeInput("TXN200", "CREDIT|CUR3001|75000",
                baseTime.plusSeconds(900));

            System.out.println("Inner join results:");
            System.out.println("  Records in output: " + outputTopic.getQueueSize());
            while (!outputTopic.isEmpty()) {
                System.out.println("  " + outputTopic.readValue());
            }
            System.out.println("  TXN100: matched (within 5-min window)");
            System.out.println("  TXN200: no output (outside 5-min window)");
        }
    }
}
TEXT
Expected output:
Inner join results:
  Records in output: 1
  MATCHED|DEBIT|SAV2001|50000|CREDIT|SAV2001|50000
  TXN100: matched (within 5-min window)
  TXN200: no output (outside 5-min window)

What should a senior or principal engineer know?

At the senior level, you need to understand the performance implications of each join type and how to choose between them.

  • Inner join creates the least state because it only stores records that have a chance of matching. It is the most efficient choice when you need strict matching.
  • Outer join creates the most state because every record from both sides must be stored for the window duration. In high-throughput systems, this can cause large RocksDB state stores.
  • Window size directly affects memory usage. A 10-minute window stores all records from both sides for 10 minutes. A 1-hour window stores 6 times more data.
  • Spurious results: in KStream-KStream joins, the first match emits immediately, but if another matching record arrives within the window, another result is emitted. This means you may see intermediate results before the final matched pair. Use the Suppressed API if you need only the final result.
  • Repartitioning cost: if the join key is different from the current key, Kafka Streams must repartition one or both streams, which creates internal topics and adds latency.
JAVA
package com.bank.streams;

public class SeniorJoinKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal join checklist:");
        System.out.println("  1. Inner join = least state, outer join = most state");
        System.out.println("  2. Window size directly affects RocksDB memory");
        System.out.println("  3. KStream-KStream joins produce spurious intermediate results");
        System.out.println("  4. Use Suppressed API for final-result-only semantics");
        System.out.println("  5. Repartitioning for different join keys adds internal topics");
        System.out.println("  6. Co-partitioning is mandatory: same partition count, same key");
        System.out.println("  7. Monitor join state store size via JMX metrics");
        System.out.println("  8. Consider KStream-KTable join to avoid windowing entirely");
    }
}
TEXT
Expected output:
Senior/Principal join checklist:
  1. Inner join = least state, outer join = most state
  2. Window size directly affects RocksDB memory
  3. KStream-KStream joins produce spurious intermediate results
  4. Use Suppressed API for final-result-only semantics
  5. Repartitioning for different join keys adds internal topics
  6. Co-partitioning is mandatory: same partition count, same key
  7. Monitor join state store size via JMX metrics
  8. Consider KStream-KTable join to avoid windowing entirely

What are common interview follow-up questions?

  • Can you join more than two streams? (Not directly. You chain two joins: join stream A with stream B, then join the result with stream C.)
  • What happens if the key types are different? (The join fails at runtime. Both sides must use the same key serde.)
  • Can you do a KStream-KTable left join? (Yes. Every stream record produces output. If no table entry exists for the key, the table side is null.)
  • What is a GlobalKTable join? (A GlobalKTable is replicated to every instance. It does not require co-partitioning, but it uses more memory because every instance holds the full table.)
  • How do you join on a non-key field? (You must repartition the stream using selectKey() to change the key to the join field before calling join().)

Visual Overview

DIAGRAM
 Inner Join (KStream-KStream)
 ============================

 ┌──────────┐     ┌──────────┐
 │ Debits   │     │ Credits  │
 │ (stream) │     │ (stream) │
 └────┬─────┘     └────┬─────┘
      │                │
      └───────┬────────┘
              │ join()
              ▼
    ┌───────────────────┐
    │ JoinWindows: 10m  │
    │ Both must match   │
    └─────────┬─────────┘
              │
              ▼
    ┌───────────────────┐
    │ Output: RECONCILED│
    │ (only if both     │
    │  sides present)   │
    └───────────────────┘


 Left Join
 =========

 ┌──────────┐     ┌──────────┐
 │ Debits   │     │ Credits  │
 │ (LEFT)   │     │ (RIGHT)  │
 └────┬─────┘     └────┬─────┘
      │                │
      └───────┬────────┘
              │ leftJoin()
              ▼
    ┌───────────────────┐
    │ Every debit emits │
    │ Credit = null OK  │
    └─────────┬─────────┘
              │
              ▼
    ┌───────────────────┐
    │ RECONCILED or     │
    │ PENDING_CREDIT    │
    └───────────────────┘


 Outer Join
 ==========

 ┌──────────┐     ┌──────────┐
 │ Debits   │     │ Credits  │
 │ (stream) │     │ (stream) │
 └────┬─────┘     └────┬─────┘
      │                │
      └───────┬────────┘
              │ outerJoin()
              ▼
    ┌───────────────────┐
    │ Every record from │
    │ either side emits │
    │ Other side = null │
    │ is allowed        │
    └─────────┬─────────┘
              │
              ▼
    ┌───────────────────┐
    │ RECONCILED or     │
    │ PENDING_CREDIT or │
    │ PENDING_DEBIT     │
    └───────────────────┘

Key Takeaway

Kafka Streams supports three join types for combining records from two streams or tables. An inner join is the strictest: it produces output only when both sides have a matching record within the join window. A left join always produces output for the left side, filling null for the right when no match exists. An outer join produces output for every record on either side, marking null for whichever side is missing.

In our reconciliation project at National Bank of India, we used an outer join to match debit and credit transactions by reference ID within a 10-minute window. Every debit and every credit appeared in the output, whether matched or not. The reconciliation team could see RECONCILED, PENDING_CREDIT, and PENDING_DEBIT statuses at a glance.

The most critical rule for joins is null handling. In a left join, the right value can be null. In an outer join, either value can be null. Forgetting to check for null is the number one cause of NullPointerException in join code. Always use a safe ValueJoiner that checks both sides before calling methods on them.

At the senior level, remember that join state grows with the window size: a larger window means more records stored in RocksDB. KStream-KStream joins can produce spurious intermediate results before the final match arrives. For final-result-only semantics, combine the join with the Suppressed API. Co-partitioning is mandatory for all joins: both input topics must have the same number of partitions and the same key.

Remember one sentence: inner join requires both sides, left join guarantees the left side, outer join guarantees both sides, and all three require co-partitioning and null-safe ValueJoiners.

---

Share
Use ← → arrow keys to navigate