Definition
- Bulk data copying -- the destination broker fetches the entire partition log from the source, competing with normal traffic for disk and network.
- Throttling trade-off -- capping replication rate reduces instantaneous contention but extends how long the reassignment runs.
- Leader movement -- consumers get NOT_LEADER_FOR_PARTITION errors and must refresh metadata and reconnect.
- Cold page cache -- the new broker has none of the partition's data in memory, so reads hit disk until the cache warms.
- Coordination overhead -- ISR changes, leader elections, and metadata propagation add churn when many partitions move at once.
Imagine National Bank of India is relocating account records from one branch vault to another while the branch stays open. Moving boxes ties up the loading dock and trucks (disk/network saturation). To avoid blocking the dock, movers go slower but occupy it longer (throttling). Tellers told the ledger moved have to look up the new location (leader movement). At the new vault, clerks have not memorized where anything is (cold cache). And the manager keeps updating the master directory (coordination overhead).
Understanding the Concepts
What is a partition reassignment?
A partition reassignment moves partition replicas from one set of brokers to another. Kafka does this to rebalance load, add or drain brokers, or recover from failures. It is an online operation, but it causes transient consumer lag and latency spikes.
# Generate and execute a reassignment plan
kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
--reassignment-json-file plan.json --execute \
--throttle 50000000 # 50 MB/s cap on replication
Cause 1: Disk and network saturation from data copying
The destination broker must fetch the entire partition log from the leader, which can be gigabytes. This competes with normal produce and consume traffic for the same disks and network.
package com.bank.kafka;
public class DataCopySaturation {
public static void main(String[] args) {
long partitionSizeGB = 50;
long networkMBps = 100; // available bandwidth
long copyTimeSec = partitionSizeGB * 1024 / networkMBps;
System.out.println("Partition size: " + partitionSizeGB + " GB");
System.out.println("Available bandwidth: " + networkMBps + " MB/s");
System.out.println("Copy time (unthrottled): " + copyTimeSec + " seconds");
System.out.println("During copy: normal produce/consume competes for same I/O");
System.out.println("Result: consumer lag spikes, producer latency increases");
}
}
Expected output:
Partition size: 50 GB
Available bandwidth: 100 MB/s
Copy time (unthrottled): 512 seconds
During copy: normal produce/consume competes for same I/O
Result: consumer lag spikes, producer latency increases
Cause 2: Throttling trades rate for duration
A replication throttle caps the bytes per second for moving data. This reduces contention but extends the duration.
package com.bank.kafka;
public class ThrottlingTradeoff {
public static void main(String[] args) {
long partitionSizeGB = 50;
long unthrottledMBps = 100;
long throttledMBps = 50;
long unthrottledSec = partitionSizeGB * 1024 / unthrottledMBps;
long throttledSec = partitionSizeGB * 1024 / throttledMBps;
System.out.println("Unthrottled: " + unthrottledSec + "s (high contention, short)");
System.out.println("Throttled (50 MB/s): " + throttledSec + "s (less contention, longer)");
System.out.println("Trade-off: intensity vs duration");
}
}
Expected output:
Unthrottled: 512s (high contention, short)
Throttled (50 MB/s): 1024s (less contention, longer)
Trade-off: intensity vs duration
Cause 3: Leader movement forces reconnection
When leaders move, consumers get NOT_LEADER_FOR_PARTITION errors, refresh metadata, and reconnect. During this window, requests fail and retry.
package com.bank.kafka;
public class LeaderMovementImpact {
public static void main(String[] args) {
System.out.println("Leader movement during reassignment:");
System.out.println(" 1. Consumer sends fetch to old leader (broker-1)");
System.out.println(" 2. Broker-1: NOT_LEADER_FOR_PARTITION");
System.out.println(" 3. Consumer refreshes metadata");
System.out.println(" 4. Discovers new leader is broker-3");
System.out.println(" 5. Reconnects to broker-3");
System.out.println(" 6. Brief latency spike for affected partitions");
System.out.println();
System.out.println("At scale: many partitions moving -> burst of metadata refreshes");
}
}
Expected output:
Leader movement during reassignment:
1. Consumer sends fetch to old leader (broker-1)
2. Broker-1: NOT_LEADER_FOR_PARTITION
3. Consumer refreshes metadata
4. Discovers new leader is broker-3
5. Reconnects to broker-3
6. Brief latency spike for affected partitions
At scale: many partitions moving -> burst of metadata refreshes
Cause 4: Cold page cache on the new broker
Kafka relies on the OS page cache for fast reads. A new broker has none of the partition's data in cache, so consumer fetches hit disk.
package com.bank.kafka;
public class ColdCacheImpact {
public static void main(String[] args) {
System.out.println("Page cache impact after leader movement:");
System.out.println(" Old broker: partition data in page cache -> memory reads");
System.out.println(" New broker: no cached data -> disk reads");
System.out.println();
System.out.println(" Memory read latency: ~100 microseconds");
System.out.println(" Disk read latency: ~10 milliseconds (100x slower)");
System.out.println();
System.out.println(" Cache warms up as consumers read data");
System.out.println(" Latency returns to normal after warm-up period");
}
}
Expected output:
Page cache impact after leader movement:
Old broker: partition data in page cache -> memory reads
New broker: no cached data -> disk reads
Memory read latency: ~100 microseconds
Disk read latency: ~10 milliseconds (100x slower)
Cache warms up as consumers read data
Latency returns to normal after warm-up period
Cause 5: Coordination overhead
Every replica movement changes the ISR, triggers metadata propagation, and possibly leader elections. At scale these add up.
How do you mitigate these causes?
Three mitigations map onto the five causes:
| Mitigation | Addresses |
|---|---|
| Monitor replication throughput | Saturation (cause 1) |
| Avoid peak traffic hours | All causes -- spare capacity absorbs impact |
| Throttle appropriately | Saturation vs duration trade-off (causes 1, 2) |
| Move in small batches | Leader movement and cold cache (causes 3, 4, 5) |
| Delay leader promotion | Cold cache (cause 4) |
# Execute with throttle and verify
kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
--reassignment-json-file plan.json --execute \
--throttle 50000000
# Monitor progress
kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
--reassignment-json-file plan.json --verify
# IMPORTANT: --verify also removes the throttle config when complete
# A forgotten throttle silently caps replication forever!
What are common mistakes?
package com.bank.kafka;
public class ReassignmentMistakes {
public static void main(String[] args) {
// WRONG: reassign all partitions at once during peak
System.out.println("WRONG: move 200 partitions at peak traffic");
System.out.println(" Result: massive lag, many NOT_LEADER errors, cold caches");
// CORRECT: small batches during off-peak with throttle
System.out.println("CORRECT: move 10 partitions at a time, off-peak, throttled");
System.out.println(" Result: minimal impact, controlled warm-up");
// WRONG: forget to run --verify after completion
System.out.println("WRONG: skip --verify after reassignment");
System.out.println(" Result: throttle stays forever, caps all future replication");
}
}
Expected output:
WRONG: move 200 partitions at peak traffic
Result: massive lag, many NOT_LEADER errors, cold caches
CORRECT: move 10 partitions at a time, off-peak, throttled
Result: minimal impact, controlled warm-up
WRONG: skip --verify after reassignment
Result: throttle stays forever, caps all future replication
What should a senior engineer know?
Delay promoting a freshly moved replica to leader until it has caught up and the cache has warmed. Raise the throttle near the end when the cluster is calmer. Always run --verify to clean up throttle configuration. Plan reassignment like any capacity-sensitive operation.
Visual Overview
Five Causes of Reassignment Lag Spikes
========================================
Reassignment triggered
│
▼
┌──────────────────────────────────────────────┐
│ 1. DISK/NETWORK SATURATION │
│ Destination copies entire partition log │
│ Competes with normal produce/consume │
├──────────────────────────────────────────────┤
│ 2. THROTTLING TRADE-OFF │
│ Lower rate = less contention │
│ But longer duration of elevated load │
├──────────────────────────────────────────────┤
│ 3. LEADER MOVEMENT │
│ NOT_LEADER_FOR_PARTITION errors │
│ Metadata refresh + reconnect │
├──────────────────────────────────────────────┤
│ 4. COLD PAGE CACHE │
│ New broker: no data in memory │
│ Reads hit disk until cache warms │
├──────────────────────────────────────────────┤
│ 5. COORDINATION OVERHEAD │
│ ISR changes, leader elections │
│ Metadata propagation to all brokers │
└──────────────────────────────────────────────┘
Mitigations:
Monitor replication throughput
Avoid peak traffic hours
Throttle appropriately
Move in small batches
Always run --verify to remove throttle
Key Takeaway
A partition reassignment is an online operation that moves replicas between brokers, but it causes transient lag and latency spikes for five reasons: disk and network saturation from bulk data copying, throttling that trades intensity for duration, leader movement forcing metadata refresh, cold page cache on the destination, and coordination overhead from ISR changes.
The mitigations map directly to these causes. Monitor replication throughput to detect saturation. Avoid peak traffic hours so spare capacity absorbs the impact. Throttle appropriately and move partitions in small batches so only a few clients see errors at once. Always run verify afterward to remove the throttle.
In my National Bank of India cluster, we saw settlement lag spike when we added brokers during peak hours. Moving in small batches off-peak with a throttle kept the impact minimal.
Remember one sentence: reassignment is safe and online but causes lag from five real causes, so plan it like any capacity-sensitive operation -- off-peak, throttled, in small batches, and always verify to clean up.
---