Definition
- Leader replica – the single replica of a partition that handles all producer writes and consumer reads, maintaining the authoritative ordered log.
- Follower replica – a passive copy of the partition that continuously fetches records from the leader to stay current, ready to be promoted if the leader fails.
- In-Sync Replica set (ISR) – the dynamic subset of replicas (leader plus caught-up followers) that have replicated all committed records within the lag window defined by
replica.lag.time.max.ms. acks=all– the producer acknowledgment setting where the leader confirms a write only after every replica in the current ISR has stored it.min.insync.replicas– a topic or broker setting that defines the minimum number of replicas that must acknowledge a write underacks=all; if the ISR shrinks below this, writes are rejected.- Unclean leader election – promoting a replica that is not in the ISR, which risks losing committed records; disabled by default in modern Kafka.
Imagine National Bank of India keeps a transaction ledger in three identical copies in three different branches for safety. One branch is the official branch (the leader) where all new entries are written and all queries are answered. The other two branches (followers) constantly copy every new entry so they match. The set of branches that are fully up to date is the in-sync set. If the official branch burns down, the bank promotes one of the fully up-to-date copies to become the new official branch, and no entries are lost.
Understanding the Concepts
What is a leader replica?
In Kafka, each partition is replicated across several brokers for fault tolerance. Among those copies, one is elected the leader. All producer writes and all consumer reads for the partition go through the leader. This single-leader design keeps ordering simple, because there is one authoritative sequence of appends.
package com.bank.replication;
public class LeaderReplicaOverview {
public static void main(String[] args) {
System.out.println("Partition: transfers-P2 (replication factor = 3)");
System.out.println();
System.out.println("Broker 1: LEADER (handles all reads and writes)");
System.out.println("Broker 2: FOLLOWER (copies from leader)");
System.out.println("Broker 3: FOLLOWER (copies from leader)");
System.out.println();
System.out.println("Producer sends to Broker 1 (leader only)");
System.out.println("Consumer reads from Broker 1 (leader only)");
}
}
Expected output:
Partition: transfers-P2 (replication factor = 3)
Broker 1: LEADER (all reads and writes)
Broker 2: FOLLOWER (copies from leader)
Broker 3: FOLLOWER (copies from leader)
What is a follower replica?
Followers do one job: they continuously fetch records from the leader and append them to their own copy of the log, trying to stay as close to the leader's latest offset as possible. Followers do not serve client reads in classic Kafka (though newer versions allow follower fetching for rack-locality). Their purpose is to be ready to take over.
package com.bank.replication;
public class FollowerReplicaBehavior {
public static void main(String[] args) {
System.out.println("Follower Replica Behavior:");
System.out.println();
System.out.println("Leader (Broker 1): offset 0..100 (latest)");
System.out.println("Follower (Broker 2): offset 0..99 (1 behind)");
System.out.println("Follower (Broker 3): offset 0..98 (2 behind)");
System.out.println();
System.out.println("Followers continuously fetch from leader:");
System.out.println(" Broker 2 fetches offset 100 -> now 0..100");
System.out.println(" Broker 3 fetches offset 99, 100 -> now 0..100");
System.out.println(" All replicas caught up -> healthy ISR");
}
}
Expected output:
Followers continuously fetch from leader to stay caught up.
All replicas caught up = healthy ISR of size 3.
What is the In-Sync Replica set (ISR)?
The ISR contains the leader plus every follower that is caught up within the allowed lag, governed by replica.lag.time.max.ms. If a follower falls behind because its broker is slow or network-partitioned, the leader removes it from the ISR. When the lagging follower catches up again, it rejoins. The ISR is a dynamic set that shrinks and grows with the health of the followers.
package com.bank.replication;
public class IsrDynamics {
public static void main(String[] args) {
System.out.println("ISR Dynamics (replica.lag.time.max.ms = 10000):");
System.out.println();
System.out.println("Healthy state:");
System.out.println(" ISR = {Broker1(L), Broker2(F), Broker3(F)}");
System.out.println(" All caught up -> ISR size = 3");
System.out.println();
System.out.println("Broker 3 becomes slow (disk I/O spike):");
System.out.println(" Broker 3 does not fetch for 15 seconds");
System.out.println(" Leader removes Broker 3 from ISR");
System.out.println(" ISR = {Broker1(L), Broker2(F)} -> size = 2");
System.out.println();
System.out.println("Broker 3 recovers and catches up:");
System.out.println(" ISR = {Broker1(L), Broker2(F), Broker3(F)} -> size = 3");
}
}
Expected output:
ISR shrinks when follower falls behind, grows when it catches up.
Dynamic set reflecting replica health.
How does the ISR interact with producer acks?
With acks=all, the leader only acknowledges a write after every replica in the current ISR has copied it. Combined with min.insync.replicas, this gives a strong durability guarantee: if min.insync.replicas=2 and the ISR shrinks below two, the leader rejects writes rather than risk accepting data that could be lost.
package com.bank.config;
import java.util.Properties;
public class DurableProducerConfig {
public static Properties createConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
// Strongest durability: wait for ALL ISR replicas
props.put("acks", "all");
props.put("enable.idempotence", "true");
props.put("retries", "3");
System.out.println("Producer: acks=all (waits for all ISR members)");
System.out.println("Topic: min.insync.replicas=2");
System.out.println("-> Write confirmed on at least 2 brokers");
return props;
}
public static void main(String[] args) {
createConfig();
}
}
Expected output:
Producer: acks=all (waits for all ISR members)
Topic: min.insync.replicas=2
-> Write confirmed on at least 2 brokers
Scenario: ISR = {Broker1, Broker2, Broker3}
Write -> Broker1 stores, Broker2 stores, Broker3 stores -> ACK
Scenario: ISR = {Broker1, Broker2} (Broker3 fell out)
Write -> Broker1 stores, Broker2 stores -> ACK (min.insync.replicas=2 met)
Scenario: ISR = {Broker1} (both followers fell out)
Write -> REJECTED (min.insync.replicas=2 NOT met)
NotEnoughReplicasException thrown
How does leader election work?
When the current leader's broker fails, the controller picks a new leader from the ISR, because only ISR members have all committed records. Choosing a leader from the ISR means no committed data is lost during failover.
package com.bank.replication;
public class LeaderElection {
public static void main(String[] args) {
System.out.println("Leader Election from ISR:");
System.out.println();
System.out.println("Before failure:");
System.out.println(" ISR = {Broker1(LEADER), Broker2, Broker3}");
System.out.println();
System.out.println("Broker 1 crashes:");
System.out.println(" Controller detects failure");
System.out.println(" Picks new leader from ISR: Broker 2");
System.out.println(" ISR = {Broker2(LEADER), Broker3}");
System.out.println(" No data lost (Broker 2 had all committed records)");
System.out.println();
System.out.println("Broker 1 recovers:");
System.out.println(" Joins as follower, catches up");
System.out.println(" ISR = {Broker2(LEADER), Broker3, Broker1}");
}
}
Expected output:
Broker 1 crashes -> Controller elects Broker 2 (ISR member)
No data lost. Broker 1 recovers as follower.
What is unclean leader election?
If you enable unclean.leader.election.enable=true, Kafka may elect a leader from outside the ISR when no ISR replica is available. This trades durability for availability: the new leader may be missing some committed records, causing data loss. This is normally disabled in banking systems.
package com.bank.replication;
// WRONG for banking: unclean leader election enabled.
// An out-of-sync replica becomes leader -> committed records lost.
public class UncleanElection {
public static void main(String[] args) {
System.out.println("Unclean Leader Election:");
System.out.println();
System.out.println("ISR = {Broker1(LEADER)} (only leader in ISR)");
System.out.println("Broker 1 crashes. No ISR member available.");
System.out.println();
System.out.println("unclean.leader.election.enable=true:");
System.out.println(" Broker 3 (out of sync, missing offsets 95-100)");
System.out.println(" Promoted to leader -> offsets 95-100 LOST");
System.out.println(" 5 transfers permanently lost!");
System.out.println();
System.out.println("unclean.leader.election.enable=false (DEFAULT):");
System.out.println(" Partition stays offline until a synced replica returns");
System.out.println(" No data loss, but partition unavailable");
System.out.println(" For banking: ALWAYS keep this false");
}
}
WRONG (banking): unclean election = data loss risk
CORRECT (banking): unclean election disabled = no data loss
How is the ISR different from the replica set? (comparison table)
| Feature | Replica Set (AR) | In-Sync Replica Set (ISR) |
|---|---|---|
| Definition | All replicas of a partition | Only caught-up replicas |
| Size | Fixed (= replication factor) | Dynamic (shrinks and grows) |
| Includes lagging replicas | Yes | No |
| Used for acks=all | No | Yes (must acknowledge) |
| Used for leader election | No | Yes (only ISR members) |
| Monitoring | Under-replicated partitions | ISR-shrink alerts |
package com.bank.replication;
public class ReplicaSetVsIsr {
public static void main(String[] args) {
System.out.println("Replica Set (AR) vs ISR:");
System.out.println();
System.out.println("Replication Factor = 3");
System.out.println("AR (All Replicas) = {Broker1, Broker2, Broker3} (always 3)");
System.out.println();
System.out.println("Healthy: ISR = {Broker1, Broker2, Broker3} (3)");
System.out.println("Broker3 slow: ISR = {Broker1, Broker2} (2)");
System.out.println("Broker2 slow: ISR = {Broker1} (1)");
System.out.println();
System.out.println("AR stays 3. ISR shrinks to 1.");
System.out.println("acks=all writes to ISR members only.");
}
}
Expected output:
AR = always 3 (fixed). ISR = dynamic (1 to 3 based on health).
What should a senior engineer know?
The senior-level nuance is the three-way tension between durability, availability, and latency. Requiring acks=all with min.insync.replicas=2 on a replication factor of three means you can tolerate one broker failure and still accept writes, while guaranteeing every acknowledged write is on at least two brokers. Monitor ISR shrink events closely because a chronically shrinking ISR signals an unhealthy broker.
package com.bank.senior;
public class IsrBestPractices {
public static void main(String[] args) {
System.out.println("Banking ISR Configuration:");
System.out.println(" replication.factor = 3");
System.out.println(" min.insync.replicas = 2");
System.out.println(" acks = all");
System.out.println(" unclean.leader.election.enable = false");
System.out.println();
System.out.println("Guarantees:");
System.out.println(" Every confirmed write on >= 2 brokers");
System.out.println(" Tolerates 1 broker failure (writes continue)");
System.out.println(" 2 broker failures: writes rejected (safety first)");
System.out.println();
System.out.println("Monitor:");
System.out.println(" ISR-shrink events -> unhealthy broker");
System.out.println(" Under-replicated partitions -> replication lag");
}
}
Expected output:
Banking config: RF=3, min.insync=2, acks=all, unclean=false
Tolerates 1 broker failure. 2 failures -> writes rejected.
Visual Overview
Leader, Followers, and ISR
===========================
Partition: transfers-P2 (RF=3)
┌──────────────────────┐
│ Broker 1 (LEADER) │
│ Offset: 0..100 │◀── Producer writes here
│ Serves reads/writes │──▶ Consumer reads here
└──────────┬───────────┘
│ Followers fetch
┌────────┴────────┐
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Broker 2 (F) │ │ Broker 3 (F) │
│ Offset: 0..100│ │ Offset: 0..98 │
│ IN SYNC │ │ LAGGING │
└──────────────┘ └──────────────┘
ISR = {Broker1, Broker2} (Broker3 fell out)
acks=all write:
Producer -> Broker1 stores -> Broker2 stores -> ACK
(Broker3 not in ISR, not waited for)
Leader failure:
Broker1 crashes -> Controller elects Broker2 (ISR member)
ISR = {Broker2(new LEADER)}
No committed data lost.
Key Takeaway
In Kafka, every partition is replicated across several brokers, with one elected as the leader and the rest as followers. The leader handles all reads and writes, keeping ordering simple because there is a single authoritative log. Followers continuously fetch records from the leader to stay caught up, and their only job is to be ready to take over if the leader fails.
The ISR is the heart of Kafka's durability model. It is the leader plus every follower that is caught up within the lag window. With acks=all, the leader only acknowledges a write after every ISR member has stored it, and min.insync.replicas sets the floor so the leader refuses writes if the ISR shrinks too small. When the leader's broker dies, the controller elects a new leader from the ISR, ensuring no committed data is lost.
At National Bank of India, transfer topics run with replication factor three, acks=all, and min.insync.replicas=2. A committed transfer always lives on at least two brokers, the system tolerates one broker failure, and if the leader crashes a caught-up replica takes over with no lost transfers. Unclean leader election is disabled because promoting an out-of-sync replica could lose committed transfers.
Remember one sentence: the ISR is the set of caught-up replicas that acknowledges writes under acks=all and provides the candidates for leader election, making it the foundation of Kafka's durability.
---