How many partitions should a topic have, what are the real constraints, and why can you never decrease partition count?

• Partition count -- sets the maximum consumer parallelism for a topic because within a consumer group each partition is consumed by at most one consumer.
• Sizing formula -- target throughput divided by per-partition throughput, with headroom for future scaling, gives the starting number.
• Broker constraints -- each partition replica is a set of files, so more partitions mean more open file handles, memory for indexes, slower rebalances, and longer recovery after failures.
• Increase only -- you can add partitions with kafka-topics --alter but can never decrease them, because reducing partitions would break the hash(key) % numPartitions mapping and strand data.
• Over-partitioning cost -- too many partitions hurt end-to-end latency, controller operations, and recovery time, so generous does not mean unlimited.

What is a Kafka partition and why does count matter?

A Kafka topic is divided into partitions, and each partition is an ordered, immutable sequence of records. The partition count matters because it caps consumer parallelism. In one consumer group a partition is owned by exactly one consumer, so a topic with six partitions can have at most six actively consuming instances in a group. A seventh consumer sits idle. This means the partition count is the ceiling on how many instances can consume in parallel.

In a banking context, the transactions topic at National Bank of India might receive thousands of fund transfer events per second. If that topic has only two partitions, only two settlement service instances can process in parallel, regardless of how many pods Kubernetes has launched.

package com.bank.kafka;

import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class TransferProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // The topic "transfers" has 12 partitions, so up to 12 consumers can work in parallel
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            producer.send(new ProducerRecord<>("transfers", "SAV2001", "DEBIT 50000"));
            System.out.println("Sent transfer event for account SAV2001");
        }
    }
}

Expected output:
Sent transfer event for account SAV2001

How do you calculate the right partition count?

The sizing formula is practical. You estimate target throughput, say sixty megabytes per second, and the throughput a single partition sustains for your consumers, say ten megabytes per second limited by downstream processing. Dividing gives six partitions as a floor, and you add headroom, often doubling, to allow future consumer scaling without a repartition.

The same reasoning applies to message rate rather than bytes if your bottleneck is records per second.

• Floor = targetThroughput / perPartitionThroughput
• With headroom = floor * 2 (typical)
• Example = 60 MB/s / 10 MB/s = 6, doubled to 12 partitions

package com.bank.kafka;

public class PartitionCalculator {
    public static int calculate(long targetMBps, long perPartitionMBps, double headroom) {
        int floor = (int) Math.ceil((double) targetMBps / perPartitionMBps);
        int withHeadroom = (int) Math.ceil(floor * headroom);
        System.out.println("Floor partitions: " + floor);
        System.out.println("With headroom (" + headroom + "x): " + withHeadroom);
        return withHeadroom;
    }

    public static void main(String[] args) {
        // National Bank of India transfers topic sizing
        int partitions = calculate(60, 10, 2.0);
        System.out.println("Recommended partitions for transfers topic: " + partitions);
    }
}

Expected output:
Floor partitions: 6
With headroom (2.0x): 12
Recommended partitions for transfers topic: 12

What are the broker-side constraints on partition count?

Each partition replica is a set of files on a broker, so more partitions mean more open file handles and more memory for index and page cache. A broker has a practical ceiling on total partitions, often in the low thousands per broker depending on version and hardware, and the cluster has a ceiling in the tens to low hundreds of thousands.

Too many partitions also slow down controller operations, lengthen rebalances, and increase end-to-end latency because the producer spreads batches thinner so each batch fills more slowly. More partitions also increase the time to recover after a broker failure because more leader elections must happen.

| Constraint | Impact | Typical Limit |
|---|---|---|
| Open file handles | OS limit per broker | ~4000-10000 partitions per broker |
| Memory (indexes + page cache) | Broker heap and OS memory | Depends on hardware |
| Controller operations | Metadata updates slow | Tens of thousands cluster-wide |
| Rebalance time | Consumer group pauses | Proportional to partition count |
| Recovery time | Leader elections per partition | Minutes at high counts |
| End-to-end latency | Batches fill slower | Measurable above ~100 partitions |

# Check current partition count and replica distribution
kafka-topics.sh --describe --topic transfers --bootstrap-server kafka-1:9092

Topic: transfers  PartitionCount: 12  ReplicationFactor: 3
  Topic: transfers  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Topic: transfers  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
  ...

Why can you never decrease partition count?

Kafka does not allow reducing partition count for two reasons. First, Kafka would have to physically move or delete the data in the removed partitions, and there is no safe general way to merge partition logs. Second and more important, producers map a key to a partition using hash(key) % numPartitions. If you change the partition count, the same key now maps to a different partition, which breaks the guarantee that all of one key's records live in one partition in order.

package com.bank.kafka;

import org.apache.kafka.common.utils.Utils;
import java.nio.charset.StandardCharsets;

public class PartitionMappingDemo {
    public static int partition(String key, int numPartitions) {
        byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
        return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
    }

    public static void main(String[] args) {
        String accountKey = "SAV2001";

        int with6 = partition(accountKey, 6);
        int with12 = partition(accountKey, 12);

        System.out.println("Account " + accountKey + " with 6 partitions  -> partition " + with6);
        System.out.println("Account " + accountKey + " with 12 partitions -> partition " + with12);
        System.out.println("Same partition? " + (with6 == with12));
        // Rajesh's records are now split across two partitions -- ordering broken
    }
}

Expected output:
Account SAV2001 with 6 partitions  -> partition 3
Account SAV2001 with 12 partitions -> partition 9
Same partition? false

How do you add partitions in practice?

You increase partitions with the alter command. This is an online operation and does not move existing data. New records for existing keys may land in different partitions than old records.

# Increase transfers topic from 6 to 12 partitions
kafka-topics.sh --alter --topic transfers --partitions 12 \
  --bootstrap-server kafka-1:9092

# Verify the change
kafka-topics.sh --describe --topic transfers --bootstrap-server kafka-1:9092

Topic: transfers  PartitionCount: 12  ReplicationFactor: 3

The new partitions start empty. Existing data stays in the original partitions. Consumers in the group automatically pick up the new partitions on the next rebalance.

What are common mistakes when sizing partitions?

The most common mistake is over-partitioning "just to be safe." Over-partitioning hurts latency, recovery, and wastes resources. The second mistake is under-partitioning a keyed topic and then having to add partitions later, which breaks key ordering.

package com.bank.kafka;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;

public class PartitionMistakes {

    // WRONG: creating 1000 partitions "just in case" for a low-volume topic
    public static Properties wrongConfig() {
        Properties props = new Properties();
        // 1000 partitions for a topic that handles 10 messages/second
        // Wastes file handles, slows rebalances, increases latency
        System.out.println("WRONG: 1000 partitions for 10 msg/s topic");
        return props;
    }

    // CORRECT: size based on throughput with reasonable headroom
    public static Properties correctConfig() {
        Properties props = new Properties();
        // 10 msg/s at 1 msg/s per partition = 10, doubled = 20 partitions
        System.out.println("CORRECT: 20 partitions for 10 msg/s topic with 2x headroom");
        return props;
    }

    public static void main(String[] args) {
        wrongConfig();
        correctConfig();
    }
}

Expected output:
WRONG: 1000 partitions for 10 msg/s topic
CORRECT: 20 partitions for 10 msg/s topic with 2x headroom

What about keyed topics and ordering guarantees?

For keyed topics where per-key ordering is critical, such as per-account transaction streams at National Bank of India, you must treat the partition count as effectively fixed. Size it generously at design time because adding partitions later breaks key placement.

Even increasing partitions has a subtle effect: new records for an existing key may land in a different partition than old ones. If strict per-key ordering across all time matters, you plan the partition count carefully up front or use a custom partitioner that is stable across partition count changes.

package com.bank.kafka;

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import java.util.Map;

// Custom partitioner that maps to a fixed number of logical buckets
public class StableAccountPartitioner implements Partitioner {
    private static final int LOGICAL_BUCKETS = 64;

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        int bucket = Math.floorMod(key.hashCode(), LOGICAL_BUCKETS);
        int numPartitions = cluster.partitionCountForTopic(topic);
        return bucket % numPartitions;
    }

    @Override public void close() {}
    @Override public void configure(Map<String, ?> configs) {}
}

What should a senior engineer know about partition sizing?

Over-partitioning to be safe is itself a cost, because it hurts latency and recovery and wastes resources. The better practice is to size with reasonable headroom, monitor consumer lag and broker partition counts, and add partitions deliberately when scaling, accepting that you may temporarily relax per-key ordering during the transition.

For keyed topics where ordering is sacred, such as per-account transaction streams, you treat the partition count as effectively fixed and size it generously at design time. For unkeyed topics where records are distributed round-robin, adding partitions is safe and straightforward.

| Topic Type | Partition Strategy | Can Add Later? |
|---|---|---|
| Keyed (account events) | Size generously up front | Risky -- breaks key ordering |
| Unkeyed (logs, metrics) | Start moderate, scale as needed | Safe -- no key ordering |
| Compacted (account state) | Size for read parallelism | Risky -- breaks key placement |

# Monitor partition count and lag across brokers
kafka-consumer-groups.sh --describe --group settlement-service \
  --bootstrap-server kafka-1:9092

How do you monitor and manage partition count in production?

Monitor broker-level metrics for total partitions, under-replicated partitions, and consumer lag. Alert when any broker approaches the practical ceiling. Use the reassignment tool to rebalance partitions across brokers after adding capacity.

# Check total partitions per broker
kafka-broker-api-versions.sh --bootstrap-server kafka-1:9092

# Monitor consumer lag per partition
kafka-consumer-groups.sh --describe --group settlement-service \
  --bootstrap-server kafka-1:9092

GROUP              TOPIC       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
settlement-service transfers   0          45023           45100           77
settlement-service transfers   1          44890           44950           60
...