Definition
- Default partitioner -- places a keyed record using
murmur2(key) % numPartitions, so the partition depends on the total count. - Silent remapping -- increasing from 6 to 12 changes the divisor; the same key maps to a different partition, splitting its records.
- Ordering break -- Kafka guarantees ordering only within a partition; records for one key now span two partitions, so ordering is lost.
- No warning -- the alter command succeeds instantly with no error, no warning, and no log entry about the key mapping change.
- Cannot undo -- Kafka does not allow decreasing partition count, so the only fix is a new topic and migration.
Imagine National Bank of India sorts customer letters into pigeonholes by dividing the account number by the number of pigeonholes and using the remainder. With six pigeonholes, Rajesh's letters always land in hole 3. Adding six more pigeonholes changes the remainder, so his new letters start landing in hole 9 while old unread letters sit in hole 3. Two different clerks now read Rajesh's letters out of order.
Understanding the Concepts
What is the key-to-partition mapping formula?
For a record with a non-null key, the producer computes partition = Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions. Because numPartitions is part of the formula, changing it changes the output for most keys.
package com.bank.kafka;
import org.apache.kafka.common.utils.Utils;
import java.nio.charset.StandardCharsets;
public class KeyMappingDemo {
public static int partition(String key, int numPartitions) {
byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
}
public static void main(String[] args) {
String[] accounts = {"SAV2001", "SAV3001", "SAV4001", "CUR1001"};
System.out.println("Key-to-partition mapping before and after repartition:");
System.out.println(String.format("%-10s %s %s %s",
"Account", "6 partitions", "12 partitions", "Changed?"));
for (String account : accounts) {
int p6 = partition(account, 6);
int p12 = partition(account, 12);
boolean changed = p6 != p12;
System.out.println(String.format("%-10s partition %-4d partition %-4d %s",
account, p6, p12, changed ? "YES -- BROKEN" : "no"));
}
}
}
Expected output:
Key-to-partition mapping before and after repartition:
Account 6 partitions 12 partitions Changed?
SAV2001 partition 3 partition 9 YES -- BROKEN
SAV3001 partition 1 partition 7 YES -- BROKEN
SAV4001 partition 5 partition 5 no
CUR1001 partition 2 partition 8 YES -- BROKEN
What ordering guarantee breaks?
Kafka's only ordering guarantee is within a single partition. Before the change, every event for account SAV2001 went to one partition. After the increase, old events are in the original partition and new events land in a different partition. Two consumers read these independently and concurrently.
package com.bank.kafka;
public class OrderingBreakScenario {
public static void main(String[] args) {
System.out.println("Before repartition (6 partitions):");
System.out.println(" SAV2001 all events -> partition 3 (ordered)");
System.out.println(" Consumer reads: DEPOSIT, then DEBIT (correct order)");
System.out.println();
System.out.println("After repartition (12 partitions):");
System.out.println(" SAV2001 old events -> partition 3 (still there)");
System.out.println(" SAV2001 new events -> partition 9 (new mapping)");
System.out.println(" Consumer A reads partition 3: old DEPOSIT");
System.out.println(" Consumer B reads partition 9: new DEBIT (arrives first!)");
System.out.println(" Result: DEBIT processed before DEPOSIT -> WRONG BALANCE");
}
}
Expected output:
Before repartition (6 partitions):
SAV2001 all events -> partition 3 (ordered)
Consumer reads: DEPOSIT, then DEBIT (correct order)
After repartition (12 partitions):
SAV2001 old events -> partition 3 (still there)
SAV2001 new events -> partition 9 (new mapping)
Consumer A reads partition 3: old DEPOSIT
Consumer B reads partition 9: new DEBIT (arrives first!)
Result: DEBIT processed before DEPOSIT -> WRONG BALANCE
Why is this change silent and dangerous?
Kafka does not throw an error, log a warning, or reject the operation. The alter command succeeds instantly. Throughput might even improve. The only symptom is subtle data correctness bugs that appear later.
# This command succeeds instantly with no warning
kafka-topics.sh --alter --topic transfers --partitions 12 \
--bootstrap-server kafka-1:9092
# No error, no warning about key remapping
# Everything looks healthy -- throughput improves
# But per-key ordering is silently broken
How do you safely increase parallelism for keyed topics?
The cleanest option is to create a new topic with the target partition count, migrate producers and consumers, and drain the old topic.
package com.bank.kafka;
public class SafeRepartitionPlan {
public static void main(String[] args) {
System.out.println("Safe repartition for keyed topics:");
System.out.println(" 1. Create new topic: transfers-v2 with 12 partitions");
System.out.println(" 2. Deploy consumers that read both transfers and transfers-v2");
System.out.println(" 3. Switch producers to write to transfers-v2");
System.out.println(" 4. Drain transfers (old topic) to completion");
System.out.println(" 5. Remove old topic subscription from consumers");
System.out.println();
System.out.println("Result: all records for SAV2001 are in one partition of transfers-v2");
System.out.println(" No key is split across old and new partitions");
}
}
Expected output:
Safe repartition for keyed topics:
1. Create new topic: transfers-v2 with 12 partitions
2. Deploy consumers that read both transfers and transfers-v2
3. Switch producers to write to transfers-v2
4. Drain transfers (old topic) to completion
5. Remove old topic subscription from consumers
Result: all records for SAV2001 are in one partition of transfers-v2
No key is split across old and new partitions
What about custom partitioners for stability?
A custom partitioner that hashes into a fixed number of logical buckets keeps placement stable across partition count changes.
package com.bank.kafka;
import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import java.util.Map;
public class StablePartitioner implements Partitioner {
private static final int LOGICAL_BUCKETS = 64;
@Override
public int partition(String topic, Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster) {
int bucket = Math.floorMod(key.hashCode(), LOGICAL_BUCKETS);
int numPartitions = cluster.partitionCountForTopic(topic);
return bucket % numPartitions;
}
@Override public void close() {}
@Override public void configure(Map<String, ?> configs) {}
}
When is adding partitions safe?
The one case where adding partitions is safe is when records have null keys. They are distributed round-robin and carry no per-key ordering expectation.
| Topic Type | Adding Partitions | Risk |
|---|---|---|
| Keyed (account events) | Breaks key ordering | HIGH |
| Null keys (logs, metrics) | Safe, round-robin | NONE |
| Compacted (account state) | Breaks key placement | HIGH |
| Kafka Streams internal | Invalidates state stores | HIGH |
What about defensive consumer-side guards?
Add a per-account expected sequence number on the consumer, so if ordering does break, the system rejects the out-of-order event instead of silently corrupting data.
package com.bank.kafka;
import java.util.HashMap;
import java.util.Map;
public class SequenceGuard {
private static final Map<String, Long> lastSequence = new HashMap<>();
public static boolean checkOrder(String accountId, long sequenceNumber) {
Long last = lastSequence.get(accountId);
if (last != null && sequenceNumber <= last) {
System.out.println("OUT OF ORDER: " + accountId
+ " expected >" + last + " got " + sequenceNumber);
return false; // Reject or alert
}
lastSequence.put(accountId, sequenceNumber);
return true;
}
public static void main(String[] args) {
checkOrder("SAV2001", 1); // OK
checkOrder("SAV2001", 2); // OK
checkOrder("SAV2001", 1); // OUT OF ORDER -- detected!
}
}
Expected output:
OUT OF ORDER: SAV2001 expected >2 got 1
What should a senior engineer know?
Increasing partitions is a one-way operation: you cannot reduce them. Plan partition counts at topic creation time based on expected peak throughput. For keyed topics where ordering matters, treat partition count as immutable. If you must increase parallelism, create a new topic with the desired partition count, migrate consumers, and replay data using a key-preserving mirror or re-keying pipeline.
package com.bank.kafka;
public class PartitionPlanningAdvice {
public static void main(String[] args) {
System.out.println("Senior advice on partition count:");
System.out.println(" 1. Partition count is effectively immutable for keyed topics");
System.out.println(" 2. Plan for peak throughput at creation time");
System.out.println(" 3. If you must change: create new topic + migrate");
System.out.println(" 4. Use consumer lag alerts to detect silent key-shuffling");
System.out.println(" 5. Document partition strategy in runbook");
}
}
Expected output:
Senior advice on partition count:
1. Partition count is effectively immutable for keyed topics
2. Plan for peak throughput at creation time
3. If you must change: create new topic + migrate
4. Use consumer lag alerts to detect silent key-shuffling
5. Document partition strategy in runbook
Visual Overview
Key-to-Partition Remapping
==========================
hash("SAV2001") = 839201
Before (6 partitions): 839201 % 6 = 3
After (12 partitions): 839201 % 12 = 9
┌──────────────────────────────────────────┐
│ Topic: transfers (6 partitions) │
│ │
│ P0 P1 P2 [P3] P4 P5 │
│ ↑ │
│ SAV2001 events (all here) │
│ Ordered: DEPOSIT, DEBIT │
└──────────────────────────────────────────┘
After --alter --partitions 12:
┌──────────────────────────────────────────────────┐
│ Topic: transfers (12 partitions) │
│ │
│ P0 P1 P2 [P3] P4 P5 P6 P7 P8 [P9]... │
│ ↑ ↑ │
│ old events new events │
│ (still here) (remapped!) │
│ │
│ Consumer A reads P3 (old DEPOSIT) │
│ Consumer B reads P9 (new DEBIT -- arrives first)│
│ Result: out-of-order processing │
└──────────────────────────────────────────────────┘
Key Takeaway
Kafka decides where a keyed record goes using the formula hash of the key modulo the number of partitions. If you increase a topic from six to twelve partitions, the divisor changes and most keys now map to a different partition. The command succeeds instantly with no warning, but per-key ordering silently breaks because records for one key are now split across two partitions.
The concrete danger is that old events for an account sit in the original partition while new events land in a different one, and two consumers read them independently, so the new events can be processed before the old ones. In banking, that is how you apply a debit before the deposit it depended on.
For keyed topics, I size partitions generously up front and treat the count as fixed. When I genuinely must scale, I create a new topic at the target count, migrate producers and consumers, and drain the old topic. I also add a defensive sequence guard on the consumer to detect ordering violations.
Remember one sentence: adding partitions changes hash(key) mod numPartitions so most keys remap, silently breaking per-key ordering with no warning from Kafka.
---