Back to Kafka Streams
Q19Question 19 of 27

What is co-partitioning and join windows in Kafka Streams

Answer: Co-partitioning is the requirement that both topics in a join have the same number of partitions and the same key-based partitioning so matching records end up on the same stream task, and a join window defines the maximum time gap allowed between two matching records.

Key Concepts
  • Co-partitioning means both input topics for a join must have the same number of partitions and use the same partitioning strategy so that records with the same key land on the same partition.
  • Without co-partitioning, records with the same key end up on different partitions and the join misses matches.
  • Kafka Streams checks co-partitioning at startup and throws a TopologyException if partition counts do not match.
  • A join window defines how far apart in time two records can be and still be considered a match in a KStream-KStream join.
  • JoinWindows.ofTimeDifferenceWithNoGrace creates a symmetric window where a record on either side can arrive up to the specified duration before or after the other.
  • KStream-KTable joins do not need a window because the table always holds the latest value per key.
  • If the join key differs from the current key, you must use selectKey() to repartition before joining.

Definition

Co-partitioning is the requirement that both topics in a join have the same number of partitions and the same key-based partitioning so matching records end up on the same stream task, and a join window defines the maximum time gap allowed between two matching records.
  • Co-partitioning means two topics share the same partition count and the same partitioning strategy (typically DefaultPartitioner) so that records with key K always land on the same partition number in both topics.
  • TopologyException is thrown at startup if Kafka Streams detects that the two topics in a join have different partition counts.
  • JoinWindows is the time-based window for KStream-KStream joins that defines how far apart two records can arrive and still be joined.
  • Repartitioning is the automatic process Kafka Streams uses when you call selectKey() before a join; it creates an internal repartition topic to redistribute records by the new key.
  • Stream task is the unit of parallelism in Kafka Streams; each task processes records from the same partition numbers of both input topics.

Think of National Bank of India with two processing queues: one for payment requests and one for payment confirmations. Co-partitioning is like ensuring that all requests from branch Mumbai go to desk 1 and all confirmations from branch Mumbai also go to desk 1. If requests go to desk 1 but confirmations go to desk 3, the clerk at desk 1 never sees the matching confirmation. A join window is the clerk's patience: they will wait up to 5 minutes for the confirmation slip. If it arrives after 5 minutes, they mark the request as unmatched.

Understanding the Concepts

What is co-partitioning?

Co-partitioning is the rule that two topics being joined must have the same number of partitions. This ensures that records with the same key are assigned to the same partition number in both topics. Kafka Streams assigns one stream task per partition, and a task reads from the same partition number in both input topics. If the partition counts differ, a record with key "SAV2001" might be on partition 2 in one topic and partition 5 in another, and the join would never find the match.

Kafka Streams checks this at startup. If the counts do not match, it throws a TopologyException before processing starts.

JAVA
package com.bank.streams;

public class CoPartitioningExplained {
    public static void main(String[] args) {
        System.out.println("Co-partitioning rule:");
        System.out.println("  Topic A (payment-requests):  3 partitions");
        System.out.println("  Topic B (payment-confirms):  3 partitions");
        System.out.println("  Key: SAV2001");
        System.out.println();
        int partitionA = Math.abs("SAV2001".hashCode()) % 3;
        int partitionB = Math.abs("SAV2001".hashCode()) % 3;
        System.out.println("  SAV2001 partition in A: " + partitionA);
        System.out.println("  SAV2001 partition in B: " + partitionB);
        System.out.println("  Same partition: " + (partitionA == partitionB));
        System.out.println();
        System.out.println("If partition counts differ (3 vs 5):");
        int wrongPartition = Math.abs("SAV2001".hashCode()) % 5;
        System.out.println("  SAV2001 partition in A (3): " + partitionA);
        System.out.println("  SAV2001 partition in B (5): " + wrongPartition);
        System.out.println("  Same partition: " + (partitionA == wrongPartition));
    }
}
TEXT
Expected output:
Co-partitioning rule:
  Topic A (payment-requests):  3 partitions
  Topic B (payment-confirms):  3 partitions
  Key: SAV2001

  SAV2001 partition in A: 1
  SAV2001 partition in B: 1
  Same partition: true

If partition counts differ (3 vs 5):
  SAV2001 partition in A (3): 1
  SAV2001 partition in B (5): 3
  Same partition: false

Why is co-partitioning required for joins?

Kafka Streams processes records locally within a stream task. A task reads from the same partition number in both input topics. For a join to find matching records, both records must be in the same task. Co-partitioning guarantees this.

Without co-partitioning, the join silently misses matches. The task processing partition 1 of the left topic does not have access to partition 3 of the right topic. There is no cross-partition communication during a join.

JAVA
package com.bank.streams;

public class WhyCoPartitioning {
    public static void main(String[] args) {
        System.out.println("Stream Task assignment:");
        System.out.println("  Task 0: reads partition 0 of both topics");
        System.out.println("  Task 1: reads partition 1 of both topics");
        System.out.println("  Task 2: reads partition 2 of both topics");
        System.out.println();
        System.out.println("If co-partitioned (both have 3 partitions):");
        System.out.println("  Key SAV2001 -> partition 1 in both topics");
        System.out.println("  Task 1 sees both records -> JOIN SUCCEEDS");
        System.out.println();
        System.out.println("If NOT co-partitioned (3 vs 5 partitions):");
        System.out.println("  Key SAV2001 -> partition 1 in topic A");
        System.out.println("  Key SAV2001 -> partition 3 in topic B");
        System.out.println("  Task 1 sees only the left record -> JOIN MISSES");
    }
}
TEXT
Expected output:
Stream Task assignment:
  Task 0: reads partition 0 of both topics
  Task 1: reads partition 1 of both topics
  Task 2: reads partition 2 of both topics

If co-partitioned (both have 3 partitions):
  Key SAV2001 -> partition 1 in both topics
  Task 1 sees both records -> JOIN SUCCEEDS

If NOT co-partitioned (3 vs 5 partitions):
  Key SAV2001 -> partition 1 in topic A
  Key SAV2001 -> partition 3 in topic B
  Task 1 sees only the left record -> JOIN MISSES

What happens without co-partitioning?

Kafka Streams validates co-partitioning at startup. If two topics have different partition counts, a TopologyException is thrown and the application does not start.

JAVA
package com.bank.streams;

public class CoPartitioningFailure {
    public static void main(String[] args) {
        // Simulating what Kafka Streams does at startup
        int partitionsA = 3;
        int partitionsB = 5;

        System.out.println("Topic payment-requests: " + partitionsA + " partitions");
        System.out.println("Topic payment-confirms: " + partitionsB + " partitions");
        System.out.println();

        if (partitionsA != partitionsB) {
            System.out.println("TopologyException: Topics not co-partitioned!");
            System.out.println("  payment-requests has " + partitionsA + " partitions");
            System.out.println("  payment-confirms has " + partitionsB + " partitions");
            System.out.println();
            System.out.println("Fix options:");
            System.out.println("  1. Recreate topics with same partition count");
            System.out.println("  2. Use through() to repartition into a new topic");
            System.out.println("  3. Use selectKey() which triggers auto-repartition");
        } else {
            System.out.println("Co-partitioning check passed");
        }
    }
}
TEXT
Expected output:
Topic payment-requests: 3 partitions
Topic payment-confirms: 5 partitions

TopologyException: Topics not co-partitioned!
  payment-requests has 3 partitions
  payment-confirms has 5 partitions

Fix options:
  1. Recreate topics with same partition count
  2. Use through() to repartition into a new topic
  3. Use selectKey() which triggers auto-repartition

What is a join window?

A join window defines the time range within which two records with the same key must arrive to be joined. It is required for KStream-KStream joins because both sides are unbounded event streams.

JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)) creates a symmetric window: a record from the left side will match a right-side record that arrives up to 5 minutes before or 5 minutes after.

KStream-KTable joins do not need a window because the table always holds the latest value per key. The stream record is joined with whatever the current table value is at that moment.

JAVA
package com.bank.streams;

import java.time.Duration;
import java.time.Instant;

public class JoinWindowExplained {
    public static void main(String[] args) {
        Duration windowSize = Duration.ofMinutes(5);
        Instant requestTime = Instant.parse("2024-01-15T10:00:00Z");

        Instant earlyConfirm = requestTime.minus(Duration.ofMinutes(3));
        Instant lateConfirm = requestTime.plus(Duration.ofMinutes(3));
        Instant tooLate = requestTime.plus(Duration.ofMinutes(7));

        System.out.println("Payment request at:    " + requestTime);
        System.out.println("Window: +/- 5 minutes");
        System.out.println();

        System.out.println("Confirmation at " + earlyConfirm + " (3 min before)");
        System.out.println("  Within window: true -> JOIN MATCH");
        System.out.println();

        System.out.println("Confirmation at " + lateConfirm + " (3 min after)");
        System.out.println("  Within window: true -> JOIN MATCH");
        System.out.println();

        System.out.println("Confirmation at " + tooLate + " (7 min after)");
        System.out.println("  Within window: false -> NO MATCH");
    }
}
TEXT
Expected output:
Payment request at:    2024-01-15T10:00:00Z
Window: +/- 5 minutes

Confirmation at 2024-01-15T09:57:00Z (3 min before)
  Within window: true -> JOIN MATCH

Confirmation at 2024-01-15T10:03:00Z (3 min after)
  Within window: true -> JOIN MATCH

Confirmation at 2024-01-15T10:07:00Z (7 min after)
  Within window: false -> NO MATCH

How do KStream-KStream, KStream-KTable, and KTable-KTable joins compare?

Each join pattern has different requirements for windows, co-partitioning, and state.

FeatureKStream-KStreamKStream-KTableKTable-KTable
Window requiredYes (JoinWindows)NoNo
Co-partitioning requiredYesYesYes
Join types supportedinner, left, outerinner, leftinner, left, outer
State storedBoth sides for window durationTable side onlyBoth table changelogs
When output is producedWhen matching records arrive within windowWhen stream record arrivesWhen either table updates
Use caseMatch two event streamsEnrich stream with lookup dataJoin two reference datasets
JAVA
package com.bank.streams;

public class JoinPatternComparison {
    public static void main(String[] args) {
        System.out.println("KStream-KStream join:");
        System.out.println("  Example: match payment requests with confirmations");
        System.out.println("  Window: required (e.g., 10 minutes)");
        System.out.println("  State: both sides stored for window duration");
        System.out.println();

        System.out.println("KStream-KTable join:");
        System.out.println("  Example: enrich transaction with account details");
        System.out.println("  Window: not needed (table holds latest value)");
        System.out.println("  State: table side stored as state store");
        System.out.println();

        System.out.println("KTable-KTable join:");
        System.out.println("  Example: join customer profile with account status");
        System.out.println("  Window: not needed (both are latest values)");
        System.out.println("  State: both sides as state stores");
    }
}
TEXT
Expected output:
KStream-KStream join:
  Example: match payment requests with confirmations
  Window: required (e.g., 10 minutes)
  State: both sides stored for window duration

KStream-KTable join:
  Example: enrich transaction with account details
  Window: not needed (table holds latest value)
  State: table side stored as state store

KTable-KTable join:
  Example: join customer profile with account status
  Window: not needed (both are latest values)
  State: both sides as state stores

What is a complete banking project for matching payments with confirmations?

This project reads payment requests from one topic and payment confirmations from another. It uses an inner join with a 5-minute window to match requests with their confirmations. The output topic receives the matched pairs.

Project structure

DIAGRAM
payment-matching-service/
├── pom.xml
└── src/
    ├── main/
    │   └── java/
    │       └── com/
    │           └── bank/
    │               └── streams/
    │                   ├── PaymentMatchingApp.java
    │                   ├── PaymentMatchingTopology.java
    │                   └── PaymentEvent.java
    └── test/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        └── PaymentMatchingTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>payment-matching-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>
    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

PaymentEvent.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class PaymentEvent {
    private String paymentId;
    private String accountId;
    private BigDecimal amount;
    private String channel;

    public PaymentEvent() {}

    public PaymentEvent(String paymentId, String accountId,
                        BigDecimal amount, String channel) {
        this.paymentId = paymentId;
        this.accountId = accountId;
        this.amount = amount;
        this.channel = channel;
    }

    public String getPaymentId() { return paymentId; }
    public String getAccountId() { return accountId; }
    public BigDecimal getAmount() { return amount; }
    public String getChannel() { return channel; }

    @Override
    public String toString() {
        return paymentId + "|" + accountId + "|" + amount + "|" + channel;
    }

    public static PaymentEvent fromString(String value) {
        String[] parts = value.split("\\|");
        return new PaymentEvent(
            parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
    }
}

PaymentMatchingTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class PaymentMatchingTopology {

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> requests = builder.stream("payment-requests");
        KStream<String, String> confirmations = builder.stream("payment-confirmations");

        // Inner join: match request with confirmation within 5 minutes
        KStream<String, String> matched = requests.join(
            confirmations,
            (requestValue, confirmValue) ->
                "CONFIRMED|" + requestValue + "|" + confirmValue,
            JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
        );

        matched.to("matched-payments");
        return builder.build();
    }
}

PaymentMatchingApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class PaymentMatchingApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-matching-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        KafkaStreams streams = new KafkaStreams(
            PaymentMatchingTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
        }));

        try {
            streams.start();
            System.out.println("Payment matching service started");
            System.out.println("  Request topic:      payment-requests");
            System.out.println("  Confirmation topic: payment-confirmations");
            System.out.println("  Output topic:       matched-payments");
            System.out.println("  Join window:        5 minutes");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

PaymentMatchingTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Instant;
import java.util.Properties;

public class PaymentMatchingTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(PaymentMatchingTopology.build(), props)) {

            TestInputTopic<String, String> requestTopic = driver.createInputTopic(
                "payment-requests", new StringSerializer(), new StringSerializer());
            TestInputTopic<String, String> confirmTopic = driver.createInputTopic(
                "payment-confirmations", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> outputTopic = driver.createOutputTopic(
                "matched-payments",
                new StringDeserializer(), new StringDeserializer());

            Instant now = Instant.now();

            // Test 1: Request and confirmation within window (2 min apart)
            requestTopic.pipeInput("PAY001",
                "PAY001|SAV2001|50000|NEFT", now);
            confirmTopic.pipeInput("PAY001",
                "PAY001|SAV2001|50000|CONFIRMED", now.plusSeconds(120));

            // Test 2: Request and confirmation outside window (10 min apart)
            requestTopic.pipeInput("PAY002",
                "PAY002|CUR3001|75000|RTGS", now.plusSeconds(300));
            confirmTopic.pipeInput("PAY002",
                "PAY002|CUR3001|75000|CONFIRMED", now.plusSeconds(900));

            System.out.println("Test results:");
            System.out.println("  Records in output: " + outputTopic.getQueueSize());
            while (!outputTopic.isEmpty()) {
                System.out.println("  " + outputTopic.readValue());
            }
            System.out.println("  PAY001: matched (2 min < 5 min window)");
            System.out.println("  PAY002: no match (10 min > 5 min window)");
        }
    }
}
TEXT
Expected output:
Test results:
  Records in output: 1
  CONFIRMED|PAY001|SAV2001|50000|NEFT|PAY001|SAV2001|50000|CONFIRMED
  PAY001: matched (2 min < 5 min window)
  PAY002: no match (10 min > 5 min window)

How to run the project

BASH
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper

# Step 2: Create co-partitioned topics (SAME partition count)
kafka-topics.sh --create --topic payment-requests --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic payment-confirmations --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic matched-payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build
cd payment-matching-service
mvn clean package

# Step 4: Run
java -cp target/payment-matching-service-1.0.0.jar com.bank.streams.PaymentMatchingApp

# Step 5: Produce requests
kafka-console-producer.sh --topic payment-requests --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# PAY001:PAY001|SAV2001|50000|NEFT

# Step 6: Produce confirmations (within 5 minutes)
kafka-console-producer.sh --topic payment-confirmations --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# PAY001:PAY001|SAV2001|50000|CONFIRMED

# Step 7: Consume matched payments
kafka-console-consumer.sh --topic matched-payments --from-beginning --bootstrap-server localhost:9092

What are common mistakes with co-partitioning and join windows?

The most common mistake is creating join topics with different partition counts.

Wrong: different partition counts

JAVA
package com.bank.streams;

public class WrongPartitionCount {
    public static void main(String[] args) {
        // Simulating creating topics with different partition counts
        int requestPartitions = 3;
        int confirmPartitions = 6;

        System.out.println("WRONG: Topics created with different partition counts");
        System.out.println("  payment-requests:      " + requestPartitions + " partitions");
        System.out.println("  payment-confirmations: " + confirmPartitions + " partitions");
        System.out.println();
        System.out.println("Result at startup:");
        System.out.println("  org.apache.kafka.streams.errors.TopologyException:");
        System.out.println("  Topics not co-partitioned:");
        System.out.println("    [payment-requests] has 3 partitions");
        System.out.println("    [payment-confirmations] has 6 partitions");
        System.out.println();
        System.out.println("The application will NOT start.");
    }
}
TEXT
Expected output:
WRONG: Topics created with different partition counts
  payment-requests:      3 partitions
  payment-confirmations: 6 partitions

Result at startup:
  org.apache.kafka.streams.errors.TopologyException:
  Topics not co-partitioned:
    [payment-requests] has 3 partitions
    [payment-confirmations] has 6 partitions

The application will NOT start.

Correct: same partition count for both topics

JAVA
package com.bank.streams;

public class CorrectPartitionCount {
    public static void main(String[] args) {
        int requestPartitions = 3;
        int confirmPartitions = 3;

        System.out.println("CORRECT: Topics created with same partition count");
        System.out.println("  payment-requests:      " + requestPartitions + " partitions");
        System.out.println("  payment-confirmations: " + confirmPartitions + " partitions");
        System.out.println();
        System.out.println("Result at startup:");
        System.out.println("  Co-partitioning check passed");
        System.out.println("  Application starts successfully");
        System.out.println();
        System.out.println("Best practice: use the same kafka-topics.sh --partitions value");
        System.out.println("for all topics that will be joined together.");
    }
}
TEXT
Expected output:
CORRECT: Topics created with same partition count
  payment-requests:      3 partitions
  payment-confirmations: 3 partitions

Result at startup:
  Co-partitioning check passed
  Application starts successfully

Best practice: use the same kafka-topics.sh --partitions value
for all topics that will be joined together.

How do you fix co-partitioning when keys differ?

If the join key is different from the current message key, you must use selectKey() to repartition the stream. Kafka Streams creates an internal repartition topic and redistributes records by the new key.

For example, payment requests are keyed by request ID, but you want to join by account ID. You call selectKey() to change the key to account ID before the join.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class RepartitionBeforeJoin {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        // Keyed by requestId
        KStream<String, String> requests = builder.stream("payment-requests");
        // Keyed by accountId
        KStream<String, String> balances = builder.stream("account-balances");

        // Change key from requestId to accountId for the join
        KStream<String, String> rekeyed = requests.selectKey(
            (requestId, value) -> {
                // value format: requestId|accountId|amount|channel
                String[] parts = value.split("\\|");
                return parts[1]; // accountId
            }
        );

        // Now both streams are keyed by accountId, join works
        KStream<String, String> enriched = rekeyed.join(
            balances,
            (requestValue, balanceValue) ->
                "ENRICHED|" + requestValue + "|balance=" + balanceValue,
            JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
        );

        enriched.to("enriched-requests");
        return builder.build();
    }

    public static void main(String[] args) {
        Topology topology = build();
        System.out.println("Topology with repartition:");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Topology with repartition:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [payment-requests])
      --> KSTREAM-KEY-SELECT-0000000002
    Processor: KSTREAM-KEY-SELECT-0000000002
      --> KSTREAM-FILTER-0000000006
    Processor: KSTREAM-FILTER-0000000006
      --> KSTREAM-SINK-0000000005
    Sink: KSTREAM-SINK-0000000005 (topic: KSTREAM-KEY-SELECT-...-repartition)

   Sub-topology: 1
    Source: KSTREAM-SOURCE-0000000007 (topics: [KSTREAM-KEY-SELECT-...-repartition])
      --> KSTREAM-WINDOWED-...
    Source: KSTREAM-SOURCE-0000000001 (topics: [account-balances])
      --> KSTREAM-WINDOWED-...
    ...

What should a senior or principal engineer know?

At the senior level, you need to understand the cost of co-partitioning decisions and join window sizing.

  • Partition count is permanent in practice: once you create a topic with N partitions, increasing it later breaks key-based ordering. Plan partition counts before the first message is produced.
  • Repartitioning doubles write traffic: selectKey() creates an internal repartition topic. Every record is written to Kafka again, then read back. This doubles the I/O for that stream.
  • Join window size vs state size: the join stores all records from both sides for the window duration. A 1-hour window on a 10,000 events/second stream stores 36 million records per side.
  • Asymmetric windows: JoinWindows.ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)) adds a 1-minute grace period for late records. Without grace, late records are dropped.
  • GlobalKTable avoids co-partitioning: a GlobalKTable is replicated to every instance. You can join any stream with it without co-partitioning, but it uses more memory.
JAVA
package com.bank.streams;

public class SeniorCoPartitioningKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal co-partitioning checklist:");
        System.out.println("  1. Plan partition count before first message");
        System.out.println("  2. selectKey() repartitioning doubles I/O");
        System.out.println("  3. Join window size = state size per partition");
        System.out.println("  4. 1-hour window at 10K/sec = 36M records per side");
        System.out.println("  5. GlobalKTable avoids co-partitioning but uses full memory");
        System.out.println("  6. Custom partitioners break co-partitioning if not identical");
        System.out.println("  7. Monitor repartition topic lag for join latency");
    }
}
TEXT
Expected output:
Senior/Principal co-partitioning checklist:
  1. Plan partition count before first message
  2. selectKey() repartitioning doubles I/O
  3. Join window size = state size per partition
  4. 1-hour window at 10K/sec = 36M records per side
  5. GlobalKTable avoids co-partitioning but uses full memory
  6. Custom partitioners break co-partitioning if not identical
  7. Monitor repartition topic lag for join latency

What are common interview follow-up questions?

  • What if I use a custom partitioner on one topic but not the other? (The join will silently miss matches because records with the same key land on different partitions.)
  • Can I change the partition count of an existing topic? (You can increase it, but existing records are not redistributed. New records may land on different partitions, breaking key ordering.)
  • What is the difference between JoinWindows and TimeWindows? (JoinWindows is for joins: it defines the allowed time gap between two matching records. TimeWindows is for aggregations: it groups records into fixed-size buckets.)
  • Does a KStream-KTable join need co-partitioning? (Yes. The table is stored locally per partition. The stream record must be on the same partition as the table entry for its key.)
  • How does Kafka Streams handle repartitioning under the hood? (It creates an internal topic named app-id-KSTREAM-KEY-SELECT-repartition, writes all records to it with the new key, then reads them back in a new sub-topology.)

Visual Overview

DIAGRAM
 Co-Partitioning: Both Topics Must Match
 ========================================

 Topic A: payment-requests (3 partitions)
 ┌──────────┬──────────┬──────────┐
 │ Part 0   │ Part 1   │ Part 2   │
 │ CUR3001  │ SAV2001  │ SAV2002  │
 └──────────┴──────────┴──────────┘

 Topic B: payment-confirmations (3 partitions)
 ┌──────────┬──────────┬──────────┐
 │ Part 0   │ Part 1   │ Part 2   │
 │ CUR3001  │ SAV2001  │ SAV2002  │
 └──────────┴──────────┴──────────┘

 Stream Tasks:
 ┌──────────────────────────────────────┐
 │ Task 0: Part 0 of A + Part 0 of B   │
 │   CUR3001 from A joins CUR3001 of B │
 ├──────────────────────────────────────┤
 │ Task 1: Part 1 of A + Part 1 of B   │
 │   SAV2001 from A joins SAV2001 of B │
 ├──────────────────────────────────────┤
 │ Task 2: Part 2 of A + Part 2 of B   │
 │   SAV2002 from A joins SAV2002 of B │
 └──────────────────────────────────────┘


 Join Window (5 minutes)
 =======================

       request                confirmation
          │                       │
  ────────┼───────────────────────┼────── time
          │<─── 5 min ───>│<─── 5 min ───>│
          │     before     │     after     │
          │                               │
          └── Records in this range match ─┘

Key Takeaway

Co-partitioning is the foundation of all joins in Kafka Streams. Both input topics must have the same number of partitions and use the same key-based partitioning strategy. Without this guarantee, records with the same key land on different partitions, and the join silently misses matches. Kafka Streams validates this at startup and throws a TopologyException if the counts differ.

A join window defines how far apart in time two records can arrive and still be matched. JoinWindows.ofTimeDifferenceWithNoGrace creates a symmetric window. A 5-minute window means a left-side record will match any right-side record that arrives up to 5 minutes before or after. KStream-KTable joins do not need a window because the table always holds the latest value per key.

In our payment matching project at National Bank of India, we used an inner join with a 5-minute window to match payment requests with confirmations. Both topics had 3 partitions, ensuring co-partitioning. Requests and confirmations arriving within the window were matched and written to the output topic. Those arriving outside the window produced no output.

At the senior level, plan partition counts before producing the first message, because changing them later breaks key ordering. Repartitioning with selectKey() doubles write traffic. Join window size directly determines state store size, so choose the smallest window that meets your SLA. For lookup-style joins where co-partitioning is impractical, consider GlobalKTable, but be aware it replicates the full table to every instance.

Remember one sentence: co-partitioning ensures matching keys land on the same partition and the same stream task, and the join window defines how long the task waits for the other side to arrive.

---

Share
Use ← → arrow keys to navigate