Definition
Every Kafka partition has one leader replica serving all reads and writes, and clients cache a map of leaders.
- Leader movement – when leadership transfers to a different broker, all clients must refresh metadata and reconnect.
- NOT_LEADER_FOR_PARTITION – the error a client gets when it sends to a broker that is no longer the leader.
- ISR shrinkage – when followers fall out of the in-sync set during churn, reducing the number of safe copies.
- Preferred leader – the first replica in a partition's assignment; Kafka periodically rebalances leadership back to it.
- Leader election rate – the JMX metric tracking how often elections happen; a rising rate is the earliest warning sign.
Imagine each type of account at National Bank of India has one designated master branch that handles all transactions, with backup branches keeping copies. If the master branch keeps changing every few minutes, customers walk into the old master and are told the account is now handled elsewhere. They waste time finding the new branch, the backlog grows during each handover, head office is constantly rewriting the master directory, and if a backup falls behind during the churn, the account has fewer safe copies. The cure is to find out why the master keeps changing, stop unnecessary reassignments, and watch how many accounts each branch handles.
Understanding the Concepts
What is partition leadership in Kafka?
For each partition, exactly one replica is the leader. All produce and consume requests go to the leader; followers only replicate. Clients cache cluster metadata that maps each partition to its current leader broker. When a leader moves, that mapping is stale and every client must refresh.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class LeaderAwareProducer {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// Retries handle transient NOT_LEADER_FOR_PARTITION
props.put(ProducerConfig.RETRIES_CONFIG, 5);
props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 200);
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("settlements", "SAV2001",
"{\"amount\":50000}")).get();
System.out.println("Settlement sent for SAV2001");
}
}
}
Settlement sent for SAV2001
How does frequent leader movement increase latency?
A client sending to the old leader gets a NOT_LEADER_FOR_PARTITION error, must fetch fresh metadata to learn the new leader, and then retry. Each cycle adds latency and shows up as a transient error. When leaders move constantly, clients spend a meaningful fraction of their time refreshing metadata and retrying instead of doing useful work.
Timeline of a leader move (partition settlements-0):
T=0s Client sends to Broker-1 (current leader)
T=0s Broker-1 fails / leader moves to Broker-2
T=0.1s Client gets NOT_LEADER_FOR_PARTITION error
T=0.3s Client refreshes metadata (network round-trip)
T=0.5s Client retries send to Broker-2 (new leader)
T=0.6s Send succeeds
Overhead: ~600ms instead of ~50ms for a normal send
How does leader movement cause consumer lag?
While a leader is moving, the affected partition is briefly unavailable for consumption. Consumers must reconnect to the new leader. During that window no progress is made, so lag ticks up each time. If leaders move frequently, these repeated small stalls accumulate.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.List;
import java.util.Properties;
public class LagMonitoringConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.METADATA_MAX_AGE_CONFIG, "30000"); // refresh metadata often during churn
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("settlements"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
records.forEach(r ->
System.out.println("Settled: " + r.key() + " partition=" + r.partition()));
}
}
}
}
Settled: SAV2001 partition=0
Settled: SAV3001 partition=2
How does leader churn risk durability?
When leadership churns, follower replicas can fall out of the in-sync replica set because they cannot keep up during the disruption. A partition that normally has three in-sync replicas can temporarily drop to two or one. If min.insync.replicas=2 and the ISR drops to one, writes are rejected, protecting durability but causing availability issues.
# Check for under-replicated partitions (ISR < replication factor)
kafka-topics.sh --describe --under-replicated-partitions \
--bootstrap-server kafka-1:9092
# Output during churn:
# Topic: settlements Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2
# (Broker-1 and Broker-3 fell out of ISR during leader movement)
What are the root causes of frequent leader movement?
Common root causes include long garbage-collection pauses that make brokers miss heartbeats and get evicted, undersized or saturated disks causing replication to fall behind, flaky network links, and misconfigured timeouts that are too short.
# Diagnose: check broker GC logs for long pauses
grep 'pause' /var/log/kafka/gc.log | tail -5
# [GC pause (G1 Evacuation Pause) 4096M->3200M(6144M), 3.2 secs]
# A 3.2-second pause exceeds the default session timeout!
# Diagnose: check under-replicated partitions trend
kafka-topics.sh --describe --under-replicated-partitions \
--bootstrap-server kafka-1:9092
# Diagnose: check leader election rate
# JMX: kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
How do you fix the root cause of broker instability?
Tune the JVM heap and garbage collector to keep pauses short. Add disk or network capacity. Correct session and heartbeat timeouts so a briefly slow broker is not evicted.
# Tune JVM GC to avoid long pauses (in kafka-server-start.sh)
export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=20"
# Tune timeouts in server.properties
# Generous enough to tolerate a brief GC, strict enough for fast failover
zookeeper.session.timeout.ms=18000
replica.lag.time.max.ms=30000
How do you avoid unnecessary preferred-leader elections?
Kafka periodically rebalances leadership back to the preferred leader. If auto rebalancing is too aggressive, it causes avoidable leader movement. Tune or disable it and run elections deliberately during calm windows.
# Broker setting: disable automatic preferred-leader rebalance
auto.leader.rebalance.enable=false
# Instead, trigger manually during a maintenance window
kafka-leader-election.sh --bootstrap-server kafka-1:9092 \
--election-type preferred --all-topic-partitions
# If you keep auto-rebalance enabled, tune the threshold
leader.imbalance.per.broker.percentage=10
leader.imbalance.check.interval.seconds=600
How do you monitor leader stability?
Watch the leader count per broker over time. An even distribution is healthy; bouncing counts reveal churn. A broker that keeps losing and regaining leadership is flapping.
# Leader distribution per broker
kafka-topics.sh --describe --bootstrap-server kafka-1:9092 | \
grep 'Leader:' | awk '{print $6}' | sort | uniq -c | sort -rn
# Expected: roughly equal counts per broker
# 120 1
# 118 2
# 122 3
# Key JMX metrics
# kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
# kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
# kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
What should a senior engineer know?
Set generous but correct session and heartbeat timeouts so a briefly slow broker is not evicted into a flap-and-recover loop. Keep GC pauses short because they are a leading cause of spurious evictions. Stagger maintenance so you never force mass leadership movement. Treat a rising leader election rate as a first-class alert because it is the earliest signal before it turns into latency and durability problems.
# Alert rules (Prometheus example)
# Alert: LeaderElectionRateHigh
# expr: rate(kafka_controller_leader_election_rate[5m]) > 0.1
# for: 5m
# labels: severity: warning
# annotations: summary: "Leader elections happening frequently"
# Alert: ISRShrinkRateHigh
# expr: rate(kafka_server_isr_shrinks_total[5m]) > 0.5
# for: 5m
# labels: severity: critical
Visual Overview
Leader Movement Impact
Normal operation:
Client ──── send() ────→ Broker-1 (Leader) ──→ Success (50ms)
After leader moves to Broker-2:
Client ──── send() ────→ Broker-1 (stale)
│
▼
NOT_LEADER_FOR_PARTITION
│
▼
Client ──── metadata refresh ────→ Controller
│
▼
Client ──── retry send() ────→ Broker-2 (new Leader) ──→ Success (600ms)
Stabilization Steps:
┌─────────────────────────────────────────────────┐
│ 1. Fix root cause (GC, disk, network, timeouts) │
│ 2. Disable auto preferred-leader rebalance │
│ 3. Run elections only in maintenance windows │
│ 4. Monitor: leader count, election rate, ISR │
└─────────────────────────────────────────────────┘
Key Takeaway
Every Kafka partition has exactly one leader replica serving all reads and writes, and clients cache a map of which broker leads each partition. When leaders move frequently, that cache is constantly stale. Clients hit NOT_LEADER_FOR_PARTITION errors, must refresh metadata and retry, which raises latency. Each transition briefly makes the partition unavailable, so consumer lag bumps up, and follower replicas can fall out of the ISR, eroding durability.
On the National Bank of India cluster, Pankaj's team saw this when one broker was under memory pressure and kept pausing for long garbage collections, missing heartbeats, getting evicted, and rejoining. Settlement producers and consumers suffered repeated errors and retries, and some partitions dropped to a single in-sync replica, which forced the team to prioritize stabilization.
Stabilizing has three parts. The most important is fixing the underlying instability: tuning the heap and garbage collector, adding capacity, or correcting timeouts. The second is avoiding unnecessary preferred-leader elections by disabling or tuning auto rebalance and running elections only during calm windows. The third is monitoring leader count per broker, leader election rate, and under-replicated partitions as early warning signals.
Remember one sentence: frequent leader movement is a symptom, and the fix is almost always to make the brokers stable and stop moving leadership unless you intend to, while watching the leader election rate as the earliest warning.
---