Definition
- Partition – an ordered, append-only log within a topic that is the unit of parallelism, ordering, and storage distribution in Kafka.
- Parallelism – the ability to process multiple records simultaneously by assigning different partitions to different consumers in a consumer group.
- Horizontal scalability – distributing a topic's partitions across many brokers so that both data and processing load scale with the cluster size.
- Ordering guarantee – Kafka's promise that records within a single partition are always read in the order they were written, controlled by the choice of message key.
- Partition key – a value attached to a record that Kafka hashes to choose the target partition, ensuring records with the same key always land in the same partition.
- Partition skew – an imbalance where one partition receives disproportionately more traffic than others due to a poorly chosen key, creating a bottleneck.
Imagine National Bank of India has one giant ledger book and only one clerk can write in it at a time: everyone queues up and it is painfully slow. Now imagine the bank splits the ledger into six books (partitions) and assigns one clerk to each book. Six clerks now work at the same time, so the bank handles far more customers. The clever rule is that all of Rajesh's entries always go into the same book, so his personal history stays in perfect order even while the overall bank runs in parallel.
Understanding the Concepts
Why do we need partitions in Kafka?
Partitions are the fundamental mechanism that gives Kafka its three core properties: parallelism, scalability, and ordering control. Without partitions, a topic would be a single log on a single broker, and only one consumer could process it at a time. With partitions, the same topic can be spread across many brokers and consumed by many consumers simultaneously.
package com.bank.config;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
import java.util.Collections;
import java.util.Properties;
public class PartitionedTopicCreator {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
try (AdminClient admin = AdminClient.create(props)) {
// 12 partitions = up to 12 parallel consumers
NewTopic topic = new NewTopic("transactions", 12, (short) 3);
admin.createTopics(Collections.singleton(topic)).all().get();
System.out.println("Topic 'transactions' created: 12 partitions");
System.out.println("Max parallel consumers per group: 12");
}
}
}
Expected output:
Topic 'transactions' created: 12 partitions
Max parallel consumers per group: 12
How do partitions enable parallelism?
A consumer group can have at most as many active consumers as there are partitions, because each partition is assigned to exactly one consumer in the group. If a topic has one partition, only one consumer can process it no matter how many you start. If it has twelve partitions, up to twelve consumers can process in parallel, giving roughly twelve times the throughput.
In Spring you simply run multiple instances of a service with the same group id, and Kafka distributes the partitions among them.
package com.bank.consumer;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class TransactionProcessor {
// Deploy 4 instances with same group id -> each gets 3 of 12 partitions
@KafkaListener(
topics = "transactions",
groupId = "settlement-group",
concurrency = "3" // 3 threads per instance
)
public void processTransaction(String event) {
System.out.println("Processing: " + event
+ " on thread " + Thread.currentThread().getName());
}
}
Expected output (4 instances, 3 threads each = 12 consumers):
Instance A, Thread 0: processes partitions 0, 1, 2
Instance A, Thread 1: processes partitions 3, 4, 5
... (but Kafka assigns at most 12 consumers for 12 partitions)
With only 1 partition:
Instance A, Thread 0: processes partition 0
Instance A, Thread 1: IDLE (no partition available)
Instance B: IDLE (no partition available)
How do partitions enable scalability?
Partitions can be spread across many brokers, so a single topic's data and load are not confined to one machine. Adding brokers and increasing partitions lets a topic scale horizontally to handle millions of events per second. Storage also scales because each broker holds only a subset of the partitions.
package com.bank.scaling;
public class PartitionScaling {
public static void main(String[] args) {
System.out.println("Partition Distribution Across Brokers:");
System.out.println();
System.out.println("3-broker cluster, 12 partitions:");
System.out.println(" Broker 1: P0, P1, P2, P3");
System.out.println(" Broker 2: P4, P5, P6, P7");
System.out.println(" Broker 3: P8, P9, P10, P11");
System.out.println();
System.out.println("6-broker cluster, 12 partitions:");
System.out.println(" Broker 1: P0, P1");
System.out.println(" Broker 2: P2, P3");
System.out.println(" Broker 3: P4, P5");
System.out.println(" Broker 4: P6, P7");
System.out.println(" Broker 5: P8, P9");
System.out.println(" Broker 6: P10, P11");
System.out.println();
System.out.println("More brokers -> each holds less data and load");
}
}
Expected output:
Partition Distribution Across Brokers:
3-broker cluster, 12 partitions:
Broker 1: P0, P1, P2, P3
Broker 2: P4, P5, P6, P7
Broker 3: P8, P9, P10, P11
6-broker cluster, 12 partitions:
Broker 1: P0, P1
Broker 2: P2, P3
...
More brokers -> each holds less data and load
How do partitions control ordering?
Kafka guarantees ordering within a partition but not across partitions. This is actually a feature, because strict global ordering would force everything through a single partition and destroy parallelism. Instead you pick a key that defines your ordering requirement. In banking, keying by account id means all events for one account are ordered, which is usually exactly what you need.
package com.bank.ordering;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class OrderedTransferProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
public OrderedTransferProducer(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishTransfer(String accountId, String type,
BigDecimal amount) {
String event = "{\"accountId\":\"" + accountId
+ "\",\"type\":\"" + type
+ "\",\"amount\":" + amount + "}";
// Key = accountId: all events for SAV2001 go to same partition
// Rajesh's debit ALWAYS comes before his balance check
kafkaTemplate.send("transactions", accountId, event);
System.out.println(type + " for " + accountId + " -> same partition");
}
public static void main(String[] args) {
// Demonstration of ordering guarantee
System.out.println("SAV2001: DEBIT -> partition 2, offset 0");
System.out.println("SAV2001: CHECK -> partition 2, offset 1");
System.out.println("SAV2001: CREDIT -> partition 2, offset 2");
System.out.println("Consumer reads: DEBIT, CHECK, CREDIT (in order)");
System.out.println();
System.out.println("SAV3002: DEBIT -> partition 5, offset 0");
System.out.println("SAV3002 and SAV2001 process IN PARALLEL");
}
}
Expected output:
SAV2001: DEBIT -> partition 2, offset 0
SAV2001: CHECK -> partition 2, offset 1
SAV2001: CREDIT -> partition 2, offset 2
Consumer reads: DEBIT, CHECK, CREDIT (in order)
SAV3002: DEBIT -> partition 5, offset 0
SAV3002 and SAV2001 process IN PARALLEL
What is partition skew and how do you avoid it?
Partition skew happens when a poorly chosen key sends disproportionate traffic to one partition. For example, if you key by transaction type and 90% of transactions are "TRANSFER", then 90% of events land in one partition while others sit idle. The solution is choosing a key with high cardinality that distributes evenly, like account id.
package com.bank.skew;
public class PartitionSkewExample {
public static void main(String[] args) {
System.out.println("BAD KEY: transaction type");
System.out.println(" key='TRANSFER' -> 90% of events -> partition 3 (HOT)");
System.out.println(" key='WITHDRAW' -> 5% of events -> partition 1 (idle)");
System.out.println(" key='DEPOSIT' -> 5% of events -> partition 0 (idle)");
System.out.println(" Result: one consumer overloaded, others idle");
System.out.println();
System.out.println("GOOD KEY: account id");
System.out.println(" key='SAV2001' -> partition 2 (balanced)");
System.out.println(" key='SAV3002' -> partition 5 (balanced)");
System.out.println(" key='SAV4003' -> partition 0 (balanced)");
System.out.println(" Result: high cardinality, even distribution");
}
}
Expected output:
BAD KEY: transaction type
key='TRANSFER' -> 90% of events -> partition 3 (HOT)
key='WITHDRAW' -> 5% of events -> partition 1 (idle)
key='DEPOSIT' -> 5% of events -> partition 0 (idle)
Result: one consumer overloaded, others idle
GOOD KEY: account id
key='SAV2001' -> partition 2 (balanced)
key='SAV3002' -> partition 5 (balanced)
key='SAV4003' -> partition 0 (balanced)
Result: high cardinality, even distribution
What happens if you have more consumers than partitions?
If there are more consumers in a group than partitions, the extra consumers sit idle because a partition can be owned by only one consumer at a time. They act as hot standbys, ready to take over if an active consumer fails.
package com.bank.consumer;
public class ConsumerVsPartitionCount {
public static void main(String[] args) {
System.out.println("Topic: transactions (3 partitions)");
System.out.println();
System.out.println("3 consumers in group:");
System.out.println(" Consumer A -> Partition 0 (active)");
System.out.println(" Consumer B -> Partition 1 (active)");
System.out.println(" Consumer C -> Partition 2 (active)");
System.out.println();
System.out.println("5 consumers in group:");
System.out.println(" Consumer A -> Partition 0 (active)");
System.out.println(" Consumer B -> Partition 1 (active)");
System.out.println(" Consumer C -> Partition 2 (active)");
System.out.println(" Consumer D -> (IDLE, no partition)");
System.out.println(" Consumer E -> (IDLE, no partition)");
System.out.println();
System.out.println("If Consumer A dies, Consumer D takes Partition 0.");
}
}
Expected output:
Topic: transactions (3 partitions)
3 consumers in group:
Consumer A -> Partition 0 (active)
Consumer B -> Partition 1 (active)
Consumer C -> Partition 2 (active)
5 consumers in group:
Consumer A -> Partition 0 (active)
Consumer B -> Partition 1 (active)
Consumer C -> Partition 2 (active)
Consumer D -> (IDLE, no partition)
Consumer E -> (IDLE, no partition)
If Consumer A dies, Consumer D takes Partition 0.
What are common mistakes with partition planning?
The most common mistake is choosing too few partitions, which caps parallelism and makes it impossible to scale consumers later. Another mistake is increasing partitions after data is already flowing, which changes key-to-partition mapping and breaks per-key ordering for existing keys.
package com.bank.mistakes;
// WRONG: Starting with 1 partition for a high-volume topic.
// You can never have more than 1 consumer, and increasing
// partitions later breaks key ordering.
public class WrongPartitionChoice {
public static void main(String[] args) {
System.out.println("WRONG: transactions topic with 1 partition");
System.out.println(" Max consumers: 1 (no parallelism)");
System.out.println(" Throughput: limited to one consumer");
System.out.println(" Increasing to 6 later: changes hash mapping");
System.out.println(" SAV2001 was on partition 0, now on partition 4");
System.out.println(" Ordering broken for existing keys!");
}
}
package com.bank.mistakes;
// CORRECT: Plan partitions with headroom from the start.
public class CorrectPartitionChoice {
public static void main(String[] args) {
System.out.println("CORRECT: transactions topic with 12 partitions");
System.out.println(" Current consumers: 4 (3 partitions each)");
System.out.println(" Peak consumers: up to 12 (1 partition each)");
System.out.println(" Headroom for scaling without repartitioning");
System.out.println(" Key ordering preserved throughout lifecycle");
}
}
WRONG: 1 partition = no parallelism, repartition breaks ordering
CORRECT: 12 partitions = room to scale, ordering preserved
What should a senior engineer know about partitions?
A senior engineer understands that choosing the partition count and the key is a real design decision, not a default. Too few partitions limits throughput. Too many partitions increase metadata overhead, leader election time, and open file handles on brokers. A good rule of thumb is to start with 2-3 times the expected peak consumer count, so you have room to scale without repartitioning.
package com.bank.senior;
public class PartitionSizingGuide {
public static void main(String[] args) {
System.out.println("Partition Sizing Guide:");
System.out.println();
System.out.println("Formula: partitions = 2-3x expected peak consumers");
System.out.println();
System.out.println("National Bank of India examples:");
System.out.println(" transactions: 12 partitions (peak 4 consumers)");
System.out.println(" audit-events: 6 partitions (peak 2 consumers)");
System.out.println(" trade-events: 24 partitions (peak 8 consumers)");
System.out.println();
System.out.println("Costs of too many partitions:");
System.out.println(" - More open file handles per broker");
System.out.println(" - Slower leader elections");
System.out.println(" - More metadata in controller");
System.out.println();
System.out.println("Costs of too few partitions:");
System.out.println(" - Cannot scale consumer group");
System.out.println(" - Repartitioning breaks key ordering");
}
}
Expected output:
Partition Sizing Guide:
Formula: partitions = 2-3x expected peak consumers
National Bank of India examples:
transactions: 12 partitions (peak 4 consumers)
audit-events: 6 partitions (peak 2 consumers)
trade-events: 24 partitions (peak 8 consumers)
Costs of too many partitions:
- More open file handles per broker
- Slower leader elections
- More metadata in controller
Costs of too few partitions:
- Cannot scale consumer group
- Repartitioning breaks key ordering
Visual Overview
Partitions: Parallelism + Ordering =================================== Topic: "transactions" (6 partitions) ┌─────────────────────────────────────────────┐ │ P0: [SAV4003] [SAV4003] [SAV4003] │ │ P1: [SAV5004] [SAV5004] │ │ P2: [SAV2001] [SAV2001] [SAV2001] [SAV2001]│ <- key=SAV2001 │ P3: [SAV6005] [SAV6005] [SAV6005] │ │ P4: [SAV7006] │ │ P5: [SAV3002] [SAV3002] [SAV3002] │ <- key=SAV3002 └─────────────────────────────────────────────┘ Consumer Group (6 consumers): ┌──────────┐ ┌──────────┐ ┌──────────┐ │Consumer A│ │Consumer B│ │Consumer C│ │ -> P0 │ │ -> P1 │ │ -> P2 │ └──────────┘ └──────────┘ └──────────┘ ┌──────────┐ ┌──────────┐ ┌──────────┐ │Consumer D│ │Consumer E│ │Consumer F│ │ -> P3 │ │ -> P4 │ │ -> P5 │ └──────────┘ └──────────┘ └──────────┘ All SAV2001 events: ordered in P2, read by Consumer C All SAV3002 events: ordered in P5, read by Consumer F 6 consumers process in PARALLEL
Key Takeaway
Partitions are the single idea that lets Kafka be both massively parallel and correctly ordered at the same time. A consumer group can have at most as many active consumers as there are partitions, so if a topic has twelve partitions, up to twelve consumers process it simultaneously, giving roughly twelve times the throughput of a single-partition topic. At National Bank of India, the transactions topic has twelve partitions, and four instances of the settlement service share them, each processing three partitions in parallel.
Scalability comes from distributing partitions across brokers. Each broker holds only a subset of the partitions, so both data and processing load scale with the cluster size. The third benefit is ordering control: Kafka guarantees strict ordering within a partition, and you control which records share a partition by choosing the message key. Keying by account id means Rajesh's debit always comes before his balance check, while Priya's events process independently on a different partition.
The senior insight is that choosing the partition count and the key is a real design decision. Too few partitions caps throughput and cannot be increased without breaking key ordering. Too many partitions add broker overhead. A good key like account id distributes load evenly across partitions while keeping per-entity ordering intact.
Remember one sentence: partitions give Kafka parallelism through multiple consumers, scalability through distributed brokers, and ordering through key-based routing, and choosing the partition count and key is one of the most important design decisions you make.
---