Definition
- Hot partition -- one key or a few keys receive disproportionate traffic, overwhelming the partition they map to while others sit idle.
- Why adding partitions fails --
hash(key) % numPartitionsis deterministic for one key; adding partitions may move it but not split it. - Key bucketing/salting -- append a bucket suffix derived from a varying field (e.g.,
BigCorp-0throughBigCorp-3) so one logical key spans multiple partitions. - Selective salting -- only salt keys that are actually hot; normal customers keep their plain key and ordering.
- Consumer trade-off -- the hot customer's records are now spread across partitions, so consumers must merge across buckets and lose strict per-logical-key ordering.
Imagine National Bank of India sorts transfer slips into counters by customer. One enterprise customer sends eighty percent of all slips, so that counter is swamped while others are nearly empty. Adding more counters does not help because the rule still sends all of that customer's slips to the same counter. The fix is to split that customer into sub-queues by tagging each slip with a sub-line number derived from the order number.
Understanding the Concepts
Why does adding partitions not fix a hot partition?
When you key a record, the partitioner computes hash(key) % numPartitions. For one specific key, that formula yields exactly one partition regardless of how many partitions exist. Adding partitions might move the key but it stays on one partition.
package com.bank.kafka;
import org.apache.kafka.common.utils.Utils;
import java.nio.charset.StandardCharsets;
public class HotPartitionDemo {
static int partition(String key, int numPartitions) {
return Utils.toPositive(Utils.murmur2(
key.getBytes(StandardCharsets.UTF_8))) % numPartitions;
}
public static void main(String[] args) {
String hotKey = "BIGCORP";
System.out.println("Key: " + hotKey);
System.out.println(" 6 partitions -> partition " + partition(hotKey, 6));
System.out.println(" 12 partitions -> partition " + partition(hotKey, 12));
System.out.println(" 24 partitions -> partition " + partition(hotKey, 24));
System.out.println(" Still ONE partition in each case -- hot spot remains");
}
}
Expected output:
Key: BIGCORP
6 partitions -> partition 2
12 partitions -> partition 8
24 partitions -> partition 20
Still ONE partition in each case -- hot spot remains
How does key bucketing/salting work?
You pick a bucket count and derive a bucket from a varying field, then append it to the key. The hot customer's traffic is split across multiple partitions.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class KeyBucketingProducer {
private static final int BUCKETS = 4;
public static String saltedKey(String customerId, String orderId) {
int bucket = Math.floorMod(orderId.hashCode(), BUCKETS);
return customerId + "-" + bucket;
}
public static void main(String[] args) {
// BigCorp's orders get distributed across 4 buckets
String[] orders = {"ORD-001", "ORD-002", "ORD-003", "ORD-004", "ORD-005"};
System.out.println("Key bucketing for BIGCORP (" + BUCKETS + " buckets):");
for (String orderId : orders) {
String key = saltedKey("BIGCORP", orderId);
System.out.println(" Order " + orderId + " -> key: " + key);
}
System.out.println(" Traffic spread across up to " + BUCKETS + " partitions");
}
}
Expected output:
Key bucketing for BIGCORP (4 buckets):
Order ORD-001 -> key: BIGCORP-1
Order ORD-002 -> key: BIGCORP-2
Order ORD-003 -> key: BIGCORP-3
Order ORD-004 -> key: BIGCORP-0
Order ORD-005 -> key: BIGCORP-1
Traffic spread across up to 4 partitions
How do you implement selective salting?
Only salt keys that are actually hot. Normal customers keep their plain key and ordering.
package com.bank.kafka;
import java.util.Set;
public class SelectiveSalting {
private static final Set<String> HOT_CUSTOMERS = Set.of("BIGCORP", "MEGABANK");
private static final int BUCKETS = 4;
public static String computeKey(String customerId, String orderId) {
if (HOT_CUSTOMERS.contains(customerId)) {
int bucket = Math.floorMod(orderId.hashCode(), BUCKETS);
return customerId + "-" + bucket; // Salted
}
return customerId; // Plain key -- ordering preserved
}
public static void main(String[] args) {
System.out.println("Hot customer: " + computeKey("BIGCORP", "ORD-001"));
System.out.println("Hot customer: " + computeKey("BIGCORP", "ORD-002"));
System.out.println("Normal customer: " + computeKey("SAV2001", "ORD-003"));
System.out.println("Normal customer: " + computeKey("SAV3001", "ORD-004"));
}
}
Expected output:
Hot customer: BIGCORP-1
Hot customer: BIGCORP-2
Normal customer: SAV2001
Normal customer: SAV3001
What is the consumer-side trade-off?
Before salting, all of a customer's records were in one partition. After salting, the hot customer's records span multiple partitions, so you lose strict ordering for that customer.
| Aspect | Before Salting | After Salting |
|---|---|---|
| Records per partition | Skewed (one overloaded) | Balanced |
| Per-customer ordering | Strict (one partition) | Lost for salted customers |
| Consumer complexity | Simple | Must merge across buckets |
| Normal customers | Unaffected | Unaffected (plain key) |
package com.bank.kafka;
public class ConsumerMergeTradeoff {
public static void main(String[] args) {
System.out.println("Consumer trade-off for salted keys:");
System.out.println(" Before: BIGCORP all events in partition 2 (ordered)");
System.out.println(" After: BIGCORP events in partitions 2, 5, 8, 11");
System.out.println();
System.out.println(" To get complete BIGCORP view:");
System.out.println(" 1. Read from all 4 buckets");
System.out.println(" 2. Buffer and sort by producer timestamp or sequence");
System.out.println(" 3. Merge into unified stream");
System.out.println();
System.out.println(" Key insight: choose salt field so per-ORDER ordering");
System.out.println(" stays in one bucket (salt by orderId)");
System.out.println(" Cross-order ordering for BIGCORP is lost but rarely needed");
}
}
Expected output:
Consumer trade-off for salted keys:
Before: BIGCORP all events in partition 2 (ordered)
After: BIGCORP events in partitions 2, 5, 8, 11
To get complete BIGCORP view:
1. Read from all 4 buckets
2. Buffer and sort by producer timestamp or sequence
3. Merge into unified stream
Key insight: choose salt field so per-ORDER ordering
stays in one bucket (salt by orderId)
Cross-order ordering for BIGCORP is lost but rarely needed
How do you roll this out without downtime?
Deploy consumers first to handle both old and new key formats, then flip producers.
package com.bank.kafka;
public class RolloutPlan {
public static void main(String[] args) {
System.out.println("Zero-downtime rollout:");
System.out.println(" 1. Deploy consumers that handle both key formats");
System.out.println(" Plain: SAV2001 (old format)");
System.out.println(" Salted: BIGCORP-2 (new format)");
System.out.println(" 2. Add BIGCORP to hot-customer config");
System.out.println(" 3. Flip producers to use salted keys for BIGCORP");
System.out.println(" 4. Old plain-keyed BIGCORP records drain naturally");
System.out.println(" No downtime, no reprocessing needed");
}
}
Expected output:
Zero-downtime rollout:
1. Deploy consumers that handle both key formats
Plain: SAV2001 (old format)
Salted: BIGCORP-2 (new format)
2. Add BIGCORP to hot-customer config
3. Flip producers to use salted keys for BIGCORP
4. Old plain-keyed BIGCORP records drain naturally
No downtime, no reprocessing needed
How do you choose the salt field?
Choose a field that varies within the hot entity so the ordering you need stays in one bucket.
| Salt Field | Ordering Preserved | Load Distribution |
|---|---|---|
| orderId | Per-order events in one bucket | Good spread |
| timestamp % N | Nothing useful | Even spread |
| Random | Nothing | Perfect spread |
What are the limitations of key bucketing?
Key bucketing is not a general solution for all ordering problems. You cannot retroactively re-key existing data already in the topic; the fix only applies to new traffic. If downstream processing requires strict global ordering for the hot customer, bucketing breaks that guarantee and you must implement merge logic. Adding too many buckets dilutes the ordering further without proportional benefit.
package com.bank.kafka;
public class BucketingLimitations {
public static void main(String[] args) {
System.out.println("Limitations of key bucketing:");
System.out.println(" 1. Existing data is NOT re-keyed (old messages stay)");
System.out.println(" 2. Strict global ordering for salted customer is lost");
System.out.println(" 3. Too many buckets: diminishing returns, more merge cost");
System.out.println(" 4. Consumer must know which keys are salted");
System.out.println(" 5. Monitoring: must track per-bucket lag, not per-customer");
System.out.println();
System.out.println("Rule of thumb: 4-8 buckets for most hot keys.");
System.out.println("More than 16 buckets rarely adds benefit.");
}
}
Expected output:
Limitations of key bucketing:
1. Existing data is NOT re-keyed (old messages stay)
2. Strict global ordering for salted customer is lost
3. Too many buckets: diminishing returns, more merge cost
4. Consumer must know which keys are salted
5. Monitoring: must track per-bucket lag, not per-customer
Rule of thumb: 4-8 buckets for most hot keys.
More than 16 buckets rarely adds benefit.
What should a senior engineer know?
Keep the hot-customer list in configuration so adding a newly hot customer is a config change, not a code change. You cannot retroactively re-key existing data; the fix applies to new traffic only.
Visual Overview
Hot Partition Before and After Salting ======================================= Before (no salting): ┌────────────────────────────────────────┐ │ P0: 100 P1: 120 P2: 8000 P3: 80 │ │ ↑ │ │ BIGCORP (80% traffic) │ │ All in one partition │ └────────────────────────────────────────┘ After (4-bucket salting for BIGCORP): ┌────────────────────────────────────────┐ │ P0: 2100 P1: 2120 P2: 2000 P3: 2080│ │ ↑ ↑ ↑ ↑ │ │ BIGCORP-0 BIGCORP-1 BIGCORP-2 BIGCORP-3 │ + normal + normal + normal + normal│ │ │ │ Load balanced across all partitions │ └────────────────────────────────────────┘ Key: customerId + "-" + (orderId.hash % 4)
Key Takeaway
A hot partition from a skewed key cannot be fixed by adding partitions because hash(key) mod numPartitions maps one key to one partition regardless of how many partitions exist. The real fix is key bucketing: append a bucket suffix derived from a varying field so the hot key spans multiple partitions.
I use selective salting so only hot customers are split while normal customers keep their plain key and ordering. The consumer trade-off is that the hot customer's records span multiple partitions, so you lose strict per-customer ordering and must merge across buckets.
In my National Bank of India corporate orders stream, we salted by order id so per-order ordering was preserved within a bucket while BigCorp's traffic spread across four partitions. We rolled it out by deploying consumers first to handle both key formats, then flipping producers.
Remember one sentence: adding partitions does not fix a hot key because one key always hashes to one partition -- use key bucketing to split the hot key across multiple partitions, and accept the merge cost on the consumer side.
---