Definition
- Sticky partitioner (no key) -- the default when no key is provided; fills one partition batch, then switches to another, spreading records evenly for throughput but providing no ordering between related records.
- Hash-of-key partitioner -- when a key is present, computes
murmur2(key) % numPartitionsso the same key always maps to the same partition, giving per-key ordering. - Custom Partitioner -- implements
org.apache.kafka.clients.producer.Partitionerand overrides thepartitionmethod with business logic, such as routing by account tier. - Ordering guarantee -- Kafka guarantees order only within a single partition; two records on different partitions have no ordering relationship.
- Explicit partition -- you can bypass the partitioner by passing a partition number directly in the
ProducerRecord, used rarely and carefully.
Think of National Bank of India directing customers to teller windows. Round-robin with no key is like a greeter who sends the next customer to whichever window is free -- all windows stay busy but one customer's two visits might go to different windows. Hash-of-key is like saying every customer always goes to the window matching the last digit of their account -- one account's transactions are always handled in order. A custom partitioner is like a VIP rule: premium accounts always go to a dedicated express window.
Understanding the Concepts
What are the three partitioning strategies?
Kafka provides three ways to decide which partition a record goes to: no key (sticky or round-robin), key-based hashing, and a custom partitioner. Each makes a different trade-off between ordering and load distribution.
package com.bank.producer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class PartitionStrategies {
public static void main(String[] args) {
// Strategy 1: No key -- sticky/round-robin
ProducerRecord<String, String> noKey =
new ProducerRecord<>("audit-log", null, "{\"event\":\"login\"}");
// Strategy 2: Key-based -- hash of key
ProducerRecord<String, String> keyed =
new ProducerRecord<>("transfers", "SAV2001", "{\"amount\":50000}");
// Strategy 3: Explicit partition (bypasses partitioner)
ProducerRecord<String, String> explicit =
new ProducerRecord<>("transfers", 0, "SAV2001", "{\"amount\":50000}");
System.out.println("No key : sticky/round-robin across partitions");
System.out.println("With key : murmur2(SAV2001) % numPartitions");
System.out.println("Explicit : forced to partition 0");
}
}
No key : sticky/round-robin across partitions
With key : murmur2(SAV2001) % numPartitions
Explicit : forced to partition 0
How does the sticky partitioner work (no key)?
When a record has no key, modern Kafka uses the sticky partitioner. It fills one partition for a batch, then switches to another. This keeps batches full and throughput high while still spreading records across partitions over time. The older behavior was strict round-robin per record.
package com.bank.producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class StickyPartitionerDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("linger.ms", "10");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
for (int i = 0; i < 100; i++) {
// No key -- sticky partitioner fills current batch
producer.send(new ProducerRecord<>("audit-log", null,
"{\"event\":\"page_view\",\"seq\":" + i + "}"));
}
System.out.println("100 keyless records sent -- spread across partitions");
System.out.println("Batch efficiency is high because sticky fills each batch");
}
}
}
100 keyless records sent -- spread across partitions
Batch efficiency is high because sticky fills each batch
How does hash-of-key partitioning work?
When a key is present, the default partitioner computes murmur2(keyBytes) % numPartitions. The same key always maps to the same partition, so all records for that key are ordered. In a bank, you key by account id so a debit written before a credit is consumed before that credit.
package com.bank.producer;
import org.apache.kafka.common.utils.Utils;
public class HashPartitionDemo {
public static void main(String[] args) {
int numPartitions = 12;
String[] accounts = {"SAV2001", "SAV2002", "SAV3045", "CORP0001"};
for (String account : accounts) {
byte[] keyBytes = account.getBytes();
int partition = Math.abs(Utils.murmur2(keyBytes)) % numPartitions;
System.out.println(" " + account + " -> partition " + partition);
}
}
}
SAV2001 -> partition 7
SAV2002 -> partition 4
SAV3045 -> partition 11
CORP0001 -> partition 2
How do you implement a custom Partitioner?
package com.bank.partitioner;
import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import org.apache.kafka.common.utils.Utils;
import java.util.Map;
public class BankTierPartitioner implements Partitioner {
private static final int PREMIUM_PARTITION_COUNT = 2;
@Override
public int partition(String topic, Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster) {
int numPartitions = cluster.partitionCountForTopic(topic);
if (keyBytes == null) {
return Math.abs(Utils.murmur2(new byte[0])) % numPartitions;
}
String keyStr = key.toString();
if (keyStr.startsWith("PREMIUM-")) {
return Math.abs(Utils.murmur2(keyBytes)) % PREMIUM_PARTITION_COUNT;
}
return PREMIUM_PARTITION_COUNT +
Math.abs(Utils.murmur2(keyBytes)) % (numPartitions - PREMIUM_PARTITION_COUNT);
}
@Override public void close() { }
@Override public void configure(Map<String, ?> configs) { }
public static void main(String[] args) {
System.out.println("BankTierPartitioner:");
System.out.println(" PREMIUM-SAV2001 -> partitions 0-1 (dedicated)");
System.out.println(" SAV2001 -> partitions 2-11 (regular pool)");
}
}
BankTierPartitioner:
PREMIUM-SAV2001 -> partitions 0-1 (dedicated)
SAV2001 -> partitions 2-11 (regular pool)
How do the strategies compare? (comparison table)
| Strategy | Ordering | Distribution | Use case |
|---|---|---|---|
| No key (sticky) | None between records | Even across all partitions | Independent events like audit logs |
| Hash-of-key | Per-key ordering | Even if key has high cardinality | Account-keyed banking events |
| Custom partitioner | Business-defined | Business-defined | Premium account routing |
| Explicit partition | Per-partition ordering | Manual | Rarely used, testing |
What are common mistakes?
package com.bank.mistake;
public class PartitioningMistakes {
public static void main(String[] args) {
// WRONG: using low-cardinality key for ordering
System.out.println("WRONG: key = \"ACTIVE\" or \"INACTIVE\"");
System.out.println(" Only 2 partitions used out of 12 -- massive skew");
System.out.println();
// CORRECT: use high-cardinality key
System.out.println("CORRECT: key = accountId (millions of values)");
System.out.println(" Even spread across all 12 partitions");
System.out.println();
// WRONG: expecting global ordering across partitions
System.out.println("WRONG: expecting transfer A before transfer B");
System.out.println(" across partitions -- Kafka does NOT guarantee this");
System.out.println();
// CORRECT: global ordering requires single partition (kills parallelism)
System.out.println("CORRECT: for global order, use 1 partition (not recommended)");
}
}
WRONG: key = "ACTIVE" or "INACTIVE"
Only 2 partitions used out of 12 -- massive skew
CORRECT: key = accountId (millions of values)
Even spread across all 12 partitions
WRONG: expecting transfer A before transfer B
across partitions -- Kafka does NOT guarantee this
CORRECT: for global order, use 1 partition (not recommended)
What happens when you change the partition count?
Changing the partition count rehashes keys. A key that previously mapped to partition 3 might now map to partition 7. Records for that key produced before and after the change are on different partitions, so ordering across the boundary is not preserved.
package com.bank.partition;
import org.apache.kafka.common.utils.Utils;
public class PartitionCountChange {
public static void main(String[] args) {
String key = "SAV2001";
byte[] keyBytes = key.getBytes();
int hash = Math.abs(Utils.murmur2(keyBytes));
int partitionBefore = hash % 6;
int partitionAfter = hash % 12;
System.out.println("Key: " + key);
System.out.println(" With 6 partitions : partition " + partitionBefore);
System.out.println(" With 12 partitions : partition " + partitionAfter);
System.out.println(" Key moved! Ordering across the change is broken.");
}
}
Key: SAV2001
With 6 partitions : partition 1
With 12 partitions : partition 7
Key moved! Ordering across the change is broken.
What should a senior engineer know?
Ordering is always only within a partition, never across partitions. Keying for ordering and keeping load even are in tension, so pick the key that satisfies the ordering requirement first, then address skew. A custom partitioner is powerful but becomes production code that must be tested and maintained. The sticky partitioner is preferred over the old round-robin for keyless records because it produces fuller batches and better throughput.
Visual Overview
Partitioning Strategies
Strategy 1: No Key (Sticky)
┌─────────┐ ┌─────────────────────────┐
│ Record │ │ P0: [r1, r2, r3] batch │
│ (no key)│─────▶│ P1: [r4, r5, r6] batch │
│ │ │ P2: [r7, r8, r9] batch │
└─────────┘ └─────────────────────────┘
Even spread, no ordering
Strategy 2: Hash of Key
┌─────────────┐ ┌───────────────────────────┐
│ key=SAV2001 │ │ P7: [SAV2001, SAV2001, ..]│
│ key=SAV3045 │───▶│ P3: [SAV3045, SAV3045, ..]│
│ key=SAV1122 │ │ P0: [SAV1122, SAV1122, ..]│
└─────────────┘ └───────────────────────────┘
Per-key ordering guaranteed
Strategy 3: Custom Partitioner
┌──────────────────┐ ┌────────────────────────┐
│ PREMIUM-SAV2001 │ │ P0-P1: Premium pool │
│ SAV2001 │───▶│ P2-P11: Regular pool │
└──────────────────┘ └────────────────────────┘
Business logic routing
Key Takeaway
There are three producer partitioning strategies, and each makes a different trade-off. When a record has no key, the sticky partitioner fills batches efficiently and spreads records evenly across partitions for maximum throughput, but provides no ordering between related records. When a record has a key, the default partitioner hashes it modulo the partition count so the same key always maps to the same partition, giving per-key ordering.
A custom partitioner lets business logic override the default. At National Bank of India, Pankaj implements Kafka's Partitioner interface to route premium corporate accounts to dedicated partitions so their settlement gets guaranteed consumer capacity. Audit events are keyless and round-robin for throughput, transfer events are keyed by account id for strict per-account ordering.
The critical constraint is that ordering is only ever within a partition, never across partitions. Changing the partition count rehashes keys and moves them to different partitions, breaking ordering across the boundary. A custom partitioner is powerful but becomes production code that must be tested and maintained.
Remember one sentence: partitioning is where you encode both your correctness requirement through ordering and your performance requirement through balance, so decide the ordering need first and pick the key that satisfies it.
---