Definition
- Co-partitioning means two topics share the same partition count and the same partitioning strategy (typically DefaultPartitioner) so that records with key K always land on the same partition number in both topics.
- TopologyException is thrown at startup if Kafka Streams detects that the two topics in a join have different partition counts.
- JoinWindows is the time-based window for KStream-KStream joins that defines how far apart two records can arrive and still be joined.
- Repartitioning is the automatic process Kafka Streams uses when you call selectKey() before a join; it creates an internal repartition topic to redistribute records by the new key.
- Stream task is the unit of parallelism in Kafka Streams; each task processes records from the same partition numbers of both input topics.
Think of National Bank of India with two processing queues: one for payment requests and one for payment confirmations. Co-partitioning is like ensuring that all requests from branch Mumbai go to desk 1 and all confirmations from branch Mumbai also go to desk 1. If requests go to desk 1 but confirmations go to desk 3, the clerk at desk 1 never sees the matching confirmation. A join window is the clerk's patience: they will wait up to 5 minutes for the confirmation slip. If it arrives after 5 minutes, they mark the request as unmatched.
Understanding the Concepts
What is co-partitioning?
Co-partitioning is the rule that two topics being joined must have the same number of partitions. This ensures that records with the same key are assigned to the same partition number in both topics. Kafka Streams assigns one stream task per partition, and a task reads from the same partition number in both input topics. If the partition counts differ, a record with key "SAV2001" might be on partition 2 in one topic and partition 5 in another, and the join would never find the match.
Kafka Streams checks this at startup. If the counts do not match, it throws a TopologyException before processing starts.
package com.bank.streams;
public class CoPartitioningExplained {
public static void main(String[] args) {
System.out.println("Co-partitioning rule:");
System.out.println(" Topic A (payment-requests): 3 partitions");
System.out.println(" Topic B (payment-confirms): 3 partitions");
System.out.println(" Key: SAV2001");
System.out.println();
int partitionA = Math.abs("SAV2001".hashCode()) % 3;
int partitionB = Math.abs("SAV2001".hashCode()) % 3;
System.out.println(" SAV2001 partition in A: " + partitionA);
System.out.println(" SAV2001 partition in B: " + partitionB);
System.out.println(" Same partition: " + (partitionA == partitionB));
System.out.println();
System.out.println("If partition counts differ (3 vs 5):");
int wrongPartition = Math.abs("SAV2001".hashCode()) % 5;
System.out.println(" SAV2001 partition in A (3): " + partitionA);
System.out.println(" SAV2001 partition in B (5): " + wrongPartition);
System.out.println(" Same partition: " + (partitionA == wrongPartition));
}
}
Expected output:
Co-partitioning rule:
Topic A (payment-requests): 3 partitions
Topic B (payment-confirms): 3 partitions
Key: SAV2001
SAV2001 partition in A: 1
SAV2001 partition in B: 1
Same partition: true
If partition counts differ (3 vs 5):
SAV2001 partition in A (3): 1
SAV2001 partition in B (5): 3
Same partition: false
Why is co-partitioning required for joins?
Kafka Streams processes records locally within a stream task. A task reads from the same partition number in both input topics. For a join to find matching records, both records must be in the same task. Co-partitioning guarantees this.
Without co-partitioning, the join silently misses matches. The task processing partition 1 of the left topic does not have access to partition 3 of the right topic. There is no cross-partition communication during a join.
package com.bank.streams;
public class WhyCoPartitioning {
public static void main(String[] args) {
System.out.println("Stream Task assignment:");
System.out.println(" Task 0: reads partition 0 of both topics");
System.out.println(" Task 1: reads partition 1 of both topics");
System.out.println(" Task 2: reads partition 2 of both topics");
System.out.println();
System.out.println("If co-partitioned (both have 3 partitions):");
System.out.println(" Key SAV2001 -> partition 1 in both topics");
System.out.println(" Task 1 sees both records -> JOIN SUCCEEDS");
System.out.println();
System.out.println("If NOT co-partitioned (3 vs 5 partitions):");
System.out.println(" Key SAV2001 -> partition 1 in topic A");
System.out.println(" Key SAV2001 -> partition 3 in topic B");
System.out.println(" Task 1 sees only the left record -> JOIN MISSES");
}
}
Expected output:
Stream Task assignment:
Task 0: reads partition 0 of both topics
Task 1: reads partition 1 of both topics
Task 2: reads partition 2 of both topics
If co-partitioned (both have 3 partitions):
Key SAV2001 -> partition 1 in both topics
Task 1 sees both records -> JOIN SUCCEEDS
If NOT co-partitioned (3 vs 5 partitions):
Key SAV2001 -> partition 1 in topic A
Key SAV2001 -> partition 3 in topic B
Task 1 sees only the left record -> JOIN MISSES
What happens without co-partitioning?
Kafka Streams validates co-partitioning at startup. If two topics have different partition counts, a TopologyException is thrown and the application does not start.
package com.bank.streams;
public class CoPartitioningFailure {
public static void main(String[] args) {
// Simulating what Kafka Streams does at startup
int partitionsA = 3;
int partitionsB = 5;
System.out.println("Topic payment-requests: " + partitionsA + " partitions");
System.out.println("Topic payment-confirms: " + partitionsB + " partitions");
System.out.println();
if (partitionsA != partitionsB) {
System.out.println("TopologyException: Topics not co-partitioned!");
System.out.println(" payment-requests has " + partitionsA + " partitions");
System.out.println(" payment-confirms has " + partitionsB + " partitions");
System.out.println();
System.out.println("Fix options:");
System.out.println(" 1. Recreate topics with same partition count");
System.out.println(" 2. Use through() to repartition into a new topic");
System.out.println(" 3. Use selectKey() which triggers auto-repartition");
} else {
System.out.println("Co-partitioning check passed");
}
}
}
Expected output:
Topic payment-requests: 3 partitions
Topic payment-confirms: 5 partitions
TopologyException: Topics not co-partitioned!
payment-requests has 3 partitions
payment-confirms has 5 partitions
Fix options:
1. Recreate topics with same partition count
2. Use through() to repartition into a new topic
3. Use selectKey() which triggers auto-repartition
What is a join window?
A join window defines the time range within which two records with the same key must arrive to be joined. It is required for KStream-KStream joins because both sides are unbounded event streams.
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)) creates a symmetric window: a record from the left side will match a right-side record that arrives up to 5 minutes before or 5 minutes after.
KStream-KTable joins do not need a window because the table always holds the latest value per key. The stream record is joined with whatever the current table value is at that moment.
package com.bank.streams;
import java.time.Duration;
import java.time.Instant;
public class JoinWindowExplained {
public static void main(String[] args) {
Duration windowSize = Duration.ofMinutes(5);
Instant requestTime = Instant.parse("2024-01-15T10:00:00Z");
Instant earlyConfirm = requestTime.minus(Duration.ofMinutes(3));
Instant lateConfirm = requestTime.plus(Duration.ofMinutes(3));
Instant tooLate = requestTime.plus(Duration.ofMinutes(7));
System.out.println("Payment request at: " + requestTime);
System.out.println("Window: +/- 5 minutes");
System.out.println();
System.out.println("Confirmation at " + earlyConfirm + " (3 min before)");
System.out.println(" Within window: true -> JOIN MATCH");
System.out.println();
System.out.println("Confirmation at " + lateConfirm + " (3 min after)");
System.out.println(" Within window: true -> JOIN MATCH");
System.out.println();
System.out.println("Confirmation at " + tooLate + " (7 min after)");
System.out.println(" Within window: false -> NO MATCH");
}
}
Expected output:
Payment request at: 2024-01-15T10:00:00Z
Window: +/- 5 minutes
Confirmation at 2024-01-15T09:57:00Z (3 min before)
Within window: true -> JOIN MATCH
Confirmation at 2024-01-15T10:03:00Z (3 min after)
Within window: true -> JOIN MATCH
Confirmation at 2024-01-15T10:07:00Z (7 min after)
Within window: false -> NO MATCH
How do KStream-KStream, KStream-KTable, and KTable-KTable joins compare?
Each join pattern has different requirements for windows, co-partitioning, and state.
| Feature | KStream-KStream | KStream-KTable | KTable-KTable |
|---|---|---|---|
| Window required | Yes (JoinWindows) | No | No |
| Co-partitioning required | Yes | Yes | Yes |
| Join types supported | inner, left, outer | inner, left | inner, left, outer |
| State stored | Both sides for window duration | Table side only | Both table changelogs |
| When output is produced | When matching records arrive within window | When stream record arrives | When either table updates |
| Use case | Match two event streams | Enrich stream with lookup data | Join two reference datasets |
package com.bank.streams;
public class JoinPatternComparison {
public static void main(String[] args) {
System.out.println("KStream-KStream join:");
System.out.println(" Example: match payment requests with confirmations");
System.out.println(" Window: required (e.g., 10 minutes)");
System.out.println(" State: both sides stored for window duration");
System.out.println();
System.out.println("KStream-KTable join:");
System.out.println(" Example: enrich transaction with account details");
System.out.println(" Window: not needed (table holds latest value)");
System.out.println(" State: table side stored as state store");
System.out.println();
System.out.println("KTable-KTable join:");
System.out.println(" Example: join customer profile with account status");
System.out.println(" Window: not needed (both are latest values)");
System.out.println(" State: both sides as state stores");
}
}
Expected output:
KStream-KStream join:
Example: match payment requests with confirmations
Window: required (e.g., 10 minutes)
State: both sides stored for window duration
KStream-KTable join:
Example: enrich transaction with account details
Window: not needed (table holds latest value)
State: table side stored as state store
KTable-KTable join:
Example: join customer profile with account status
Window: not needed (both are latest values)
State: both sides as state stores
What is a complete banking project for matching payments with confirmations?
This project reads payment requests from one topic and payment confirmations from another. It uses an inner join with a 5-minute window to match requests with their confirmations. The output topic receives the matched pairs.
Project structure
payment-matching-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── PaymentMatchingApp.java
│ ├── PaymentMatchingTopology.java
│ └── PaymentEvent.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── PaymentMatchingTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>payment-matching-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
PaymentEvent.java
package com.bank.streams;
import java.math.BigDecimal;
public class PaymentEvent {
private String paymentId;
private String accountId;
private BigDecimal amount;
private String channel;
public PaymentEvent() {}
public PaymentEvent(String paymentId, String accountId,
BigDecimal amount, String channel) {
this.paymentId = paymentId;
this.accountId = accountId;
this.amount = amount;
this.channel = channel;
}
public String getPaymentId() { return paymentId; }
public String getAccountId() { return accountId; }
public BigDecimal getAmount() { return amount; }
public String getChannel() { return channel; }
@Override
public String toString() {
return paymentId + "|" + accountId + "|" + amount + "|" + channel;
}
public static PaymentEvent fromString(String value) {
String[] parts = value.split("\\|");
return new PaymentEvent(
parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
PaymentMatchingTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class PaymentMatchingTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> requests = builder.stream("payment-requests");
KStream<String, String> confirmations = builder.stream("payment-confirmations");
// Inner join: match request with confirmation within 5 minutes
KStream<String, String> matched = requests.join(
confirmations,
(requestValue, confirmValue) ->
"CONFIRMED|" + requestValue + "|" + confirmValue,
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
);
matched.to("matched-payments");
return builder.build();
}
}
PaymentMatchingApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class PaymentMatchingApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-matching-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
PaymentMatchingTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
}));
try {
streams.start();
System.out.println("Payment matching service started");
System.out.println(" Request topic: payment-requests");
System.out.println(" Confirmation topic: payment-confirmations");
System.out.println(" Output topic: matched-payments");
System.out.println(" Join window: 5 minutes");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
PaymentMatchingTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Instant;
import java.util.Properties;
public class PaymentMatchingTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(PaymentMatchingTopology.build(), props)) {
TestInputTopic<String, String> requestTopic = driver.createInputTopic(
"payment-requests", new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> confirmTopic = driver.createInputTopic(
"payment-confirmations", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> outputTopic = driver.createOutputTopic(
"matched-payments",
new StringDeserializer(), new StringDeserializer());
Instant now = Instant.now();
// Test 1: Request and confirmation within window (2 min apart)
requestTopic.pipeInput("PAY001",
"PAY001|SAV2001|50000|NEFT", now);
confirmTopic.pipeInput("PAY001",
"PAY001|SAV2001|50000|CONFIRMED", now.plusSeconds(120));
// Test 2: Request and confirmation outside window (10 min apart)
requestTopic.pipeInput("PAY002",
"PAY002|CUR3001|75000|RTGS", now.plusSeconds(300));
confirmTopic.pipeInput("PAY002",
"PAY002|CUR3001|75000|CONFIRMED", now.plusSeconds(900));
System.out.println("Test results:");
System.out.println(" Records in output: " + outputTopic.getQueueSize());
while (!outputTopic.isEmpty()) {
System.out.println(" " + outputTopic.readValue());
}
System.out.println(" PAY001: matched (2 min < 5 min window)");
System.out.println(" PAY002: no match (10 min > 5 min window)");
}
}
}
Expected output:
Test results:
Records in output: 1
CONFIRMED|PAY001|SAV2001|50000|NEFT|PAY001|SAV2001|50000|CONFIRMED
PAY001: matched (2 min < 5 min window)
PAY002: no match (10 min > 5 min window)
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create co-partitioned topics (SAME partition count)
kafka-topics.sh --create --topic payment-requests --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic payment-confirmations --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic matched-payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build
cd payment-matching-service
mvn clean package
# Step 4: Run
java -cp target/payment-matching-service-1.0.0.jar com.bank.streams.PaymentMatchingApp
# Step 5: Produce requests
kafka-console-producer.sh --topic payment-requests --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# PAY001:PAY001|SAV2001|50000|NEFT
# Step 6: Produce confirmations (within 5 minutes)
kafka-console-producer.sh --topic payment-confirmations --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# PAY001:PAY001|SAV2001|50000|CONFIRMED
# Step 7: Consume matched payments
kafka-console-consumer.sh --topic matched-payments --from-beginning --bootstrap-server localhost:9092
What are common mistakes with co-partitioning and join windows?
The most common mistake is creating join topics with different partition counts.
Wrong: different partition counts
package com.bank.streams;
public class WrongPartitionCount {
public static void main(String[] args) {
// Simulating creating topics with different partition counts
int requestPartitions = 3;
int confirmPartitions = 6;
System.out.println("WRONG: Topics created with different partition counts");
System.out.println(" payment-requests: " + requestPartitions + " partitions");
System.out.println(" payment-confirmations: " + confirmPartitions + " partitions");
System.out.println();
System.out.println("Result at startup:");
System.out.println(" org.apache.kafka.streams.errors.TopologyException:");
System.out.println(" Topics not co-partitioned:");
System.out.println(" [payment-requests] has 3 partitions");
System.out.println(" [payment-confirmations] has 6 partitions");
System.out.println();
System.out.println("The application will NOT start.");
}
}
Expected output:
WRONG: Topics created with different partition counts
payment-requests: 3 partitions
payment-confirmations: 6 partitions
Result at startup:
org.apache.kafka.streams.errors.TopologyException:
Topics not co-partitioned:
[payment-requests] has 3 partitions
[payment-confirmations] has 6 partitions
The application will NOT start.
Correct: same partition count for both topics
package com.bank.streams;
public class CorrectPartitionCount {
public static void main(String[] args) {
int requestPartitions = 3;
int confirmPartitions = 3;
System.out.println("CORRECT: Topics created with same partition count");
System.out.println(" payment-requests: " + requestPartitions + " partitions");
System.out.println(" payment-confirmations: " + confirmPartitions + " partitions");
System.out.println();
System.out.println("Result at startup:");
System.out.println(" Co-partitioning check passed");
System.out.println(" Application starts successfully");
System.out.println();
System.out.println("Best practice: use the same kafka-topics.sh --partitions value");
System.out.println("for all topics that will be joined together.");
}
}
Expected output:
CORRECT: Topics created with same partition count
payment-requests: 3 partitions
payment-confirmations: 3 partitions
Result at startup:
Co-partitioning check passed
Application starts successfully
Best practice: use the same kafka-topics.sh --partitions value
for all topics that will be joined together.
How do you fix co-partitioning when keys differ?
If the join key is different from the current message key, you must use selectKey() to repartition the stream. Kafka Streams creates an internal repartition topic and redistributes records by the new key.
For example, payment requests are keyed by request ID, but you want to join by account ID. You call selectKey() to change the key to account ID before the join.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class RepartitionBeforeJoin {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// Keyed by requestId
KStream<String, String> requests = builder.stream("payment-requests");
// Keyed by accountId
KStream<String, String> balances = builder.stream("account-balances");
// Change key from requestId to accountId for the join
KStream<String, String> rekeyed = requests.selectKey(
(requestId, value) -> {
// value format: requestId|accountId|amount|channel
String[] parts = value.split("\\|");
return parts[1]; // accountId
}
);
// Now both streams are keyed by accountId, join works
KStream<String, String> enriched = rekeyed.join(
balances,
(requestValue, balanceValue) ->
"ENRICHED|" + requestValue + "|balance=" + balanceValue,
JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
);
enriched.to("enriched-requests");
return builder.build();
}
public static void main(String[] args) {
Topology topology = build();
System.out.println("Topology with repartition:");
System.out.println(topology.describe());
}
}
Expected output:
Topology with repartition:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [payment-requests])
--> KSTREAM-KEY-SELECT-0000000002
Processor: KSTREAM-KEY-SELECT-0000000002
--> KSTREAM-FILTER-0000000006
Processor: KSTREAM-FILTER-0000000006
--> KSTREAM-SINK-0000000005
Sink: KSTREAM-SINK-0000000005 (topic: KSTREAM-KEY-SELECT-...-repartition)
Sub-topology: 1
Source: KSTREAM-SOURCE-0000000007 (topics: [KSTREAM-KEY-SELECT-...-repartition])
--> KSTREAM-WINDOWED-...
Source: KSTREAM-SOURCE-0000000001 (topics: [account-balances])
--> KSTREAM-WINDOWED-...
...
What should a senior or principal engineer know?
At the senior level, you need to understand the cost of co-partitioning decisions and join window sizing.
- Partition count is permanent in practice: once you create a topic with N partitions, increasing it later breaks key-based ordering. Plan partition counts before the first message is produced.
- Repartitioning doubles write traffic: selectKey() creates an internal repartition topic. Every record is written to Kafka again, then read back. This doubles the I/O for that stream.
- Join window size vs state size: the join stores all records from both sides for the window duration. A 1-hour window on a 10,000 events/second stream stores 36 million records per side.
- Asymmetric windows:
JoinWindows.ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1))adds a 1-minute grace period for late records. Without grace, late records are dropped. - GlobalKTable avoids co-partitioning: a GlobalKTable is replicated to every instance. You can join any stream with it without co-partitioning, but it uses more memory.
package com.bank.streams;
public class SeniorCoPartitioningKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal co-partitioning checklist:");
System.out.println(" 1. Plan partition count before first message");
System.out.println(" 2. selectKey() repartitioning doubles I/O");
System.out.println(" 3. Join window size = state size per partition");
System.out.println(" 4. 1-hour window at 10K/sec = 36M records per side");
System.out.println(" 5. GlobalKTable avoids co-partitioning but uses full memory");
System.out.println(" 6. Custom partitioners break co-partitioning if not identical");
System.out.println(" 7. Monitor repartition topic lag for join latency");
}
}
Expected output:
Senior/Principal co-partitioning checklist:
1. Plan partition count before first message
2. selectKey() repartitioning doubles I/O
3. Join window size = state size per partition
4. 1-hour window at 10K/sec = 36M records per side
5. GlobalKTable avoids co-partitioning but uses full memory
6. Custom partitioners break co-partitioning if not identical
7. Monitor repartition topic lag for join latency
What are common interview follow-up questions?
- What if I use a custom partitioner on one topic but not the other? (The join will silently miss matches because records with the same key land on different partitions.)
- Can I change the partition count of an existing topic? (You can increase it, but existing records are not redistributed. New records may land on different partitions, breaking key ordering.)
- What is the difference between JoinWindows and TimeWindows? (JoinWindows is for joins: it defines the allowed time gap between two matching records. TimeWindows is for aggregations: it groups records into fixed-size buckets.)
- Does a KStream-KTable join need co-partitioning? (Yes. The table is stored locally per partition. The stream record must be on the same partition as the table entry for its key.)
- How does Kafka Streams handle repartitioning under the hood? (It creates an internal topic named app-id-KSTREAM-KEY-SELECT-repartition, writes all records to it with the new key, then reads them back in a new sub-topology.)
Visual Overview
Co-Partitioning: Both Topics Must Match
========================================
Topic A: payment-requests (3 partitions)
┌──────────┬──────────┬──────────┐
│ Part 0 │ Part 1 │ Part 2 │
│ CUR3001 │ SAV2001 │ SAV2002 │
└──────────┴──────────┴──────────┘
Topic B: payment-confirmations (3 partitions)
┌──────────┬──────────┬──────────┐
│ Part 0 │ Part 1 │ Part 2 │
│ CUR3001 │ SAV2001 │ SAV2002 │
└──────────┴──────────┴──────────┘
Stream Tasks:
┌──────────────────────────────────────┐
│ Task 0: Part 0 of A + Part 0 of B │
│ CUR3001 from A joins CUR3001 of B │
├──────────────────────────────────────┤
│ Task 1: Part 1 of A + Part 1 of B │
│ SAV2001 from A joins SAV2001 of B │
├──────────────────────────────────────┤
│ Task 2: Part 2 of A + Part 2 of B │
│ SAV2002 from A joins SAV2002 of B │
└──────────────────────────────────────┘
Join Window (5 minutes)
=======================
request confirmation
│ │
────────┼───────────────────────┼────── time
│<─── 5 min ───>│<─── 5 min ───>│
│ before │ after │
│ │
└── Records in this range match ─┘
Key Takeaway
Co-partitioning is the foundation of all joins in Kafka Streams. Both input topics must have the same number of partitions and use the same key-based partitioning strategy. Without this guarantee, records with the same key land on different partitions, and the join silently misses matches. Kafka Streams validates this at startup and throws a TopologyException if the counts differ.
A join window defines how far apart in time two records can arrive and still be matched. JoinWindows.ofTimeDifferenceWithNoGrace creates a symmetric window. A 5-minute window means a left-side record will match any right-side record that arrives up to 5 minutes before or after. KStream-KTable joins do not need a window because the table always holds the latest value per key.
In our payment matching project at National Bank of India, we used an inner join with a 5-minute window to match payment requests with confirmations. Both topics had 3 partitions, ensuring co-partitioning. Requests and confirmations arriving within the window were matched and written to the output topic. Those arriving outside the window produced no output.
At the senior level, plan partition counts before producing the first message, because changing them later breaks key ordering. Repartitioning with selectKey() doubles write traffic. Join window size directly determines state store size, so choose the smallest window that meets your SLA. For lookup-style joins where co-partitioning is impractical, consider GlobalKTable, but be aware it replicates the full table to every instance.
Remember one sentence: co-partitioning ensures matching keys land on the same partition and the same stream task, and the join window defines how long the task waits for the other side to arrive.
---