Definition
- Fault tolerance – a system's ability to continue operating correctly and without data loss despite the failure of some of its components.
- Replication factor – the number of copies of each partition across brokers, providing redundancy so data survives broker failures.
- ISR-based leader election – promoting a new partition leader from the in-sync replica set, which guarantees the new leader has all committed records.
acks=all– the producer setting that waits for all ISR replicas to store a write before acknowledging, ensuring the data is durable.min.insync.replicas– the minimum ISR size required to accept writes, rejecting writes when the ISR is too small to guarantee durability.- Unclean leader election – promoting an out-of-sync replica when no ISR member is available, which risks data loss and is disabled for financial systems.
Fault tolerance is like National Bank of India staying open even when things break. The bank keeps three copies of each ledger drawer in three different vaults (replication factor). It only tells a customer "your deposit is confirmed" after at least two vaults have recorded it (acks and min.insync.replicas). If the main vault burns down, the bank immediately promotes one of the up-to-date copies to be the new main vault (leader election from the ISR). The customer never notices, and no record is lost.
Understanding the Concepts
What is fault tolerance?
Fault tolerance in distributed systems means two things: durability, so committed data is never lost, and availability, so the service keeps responding even when components fail. Kafka delivers both through replication, and the replication factor is the foundation.
package com.bank.ft;
public class FaultToleranceDefinition {
public static void main(String[] args) {
System.out.println("Fault Tolerance = Durability + Availability");
System.out.println();
System.out.println("Durability: committed data never lost");
System.out.println(" -> Replication + acks=all + min.insync.replicas");
System.out.println();
System.out.println("Availability: service keeps responding");
System.out.println(" -> ISR-based leader election");
System.out.println(" -> Automatic failover to surviving replicas");
}
}
Expected output:
Fault Tolerance = Durability + Availability
Durability: replication + acks + min.insync.replicas
Availability: ISR-based leader election
How does the replication factor provide redundancy?
With a replication factor of three, every partition exists as three copies on three brokers. Losing one or even two brokers still leaves a copy. If you had a replication factor of one, a single broker failure would lose that partition permanently.
package com.bank.ft;
public class RedundancyThroughReplication {
public static void main(String[] args) {
System.out.println("Replication Factor and Redundancy:");
System.out.println();
System.out.println("RF=1: [Broker1] -> 1 failure = DATA LOST");
System.out.println("RF=2: [Broker1, Broker2] -> 1 failure = safe");
System.out.println("RF=3: [B1, B2, B3] -> 2 failures = safe");
System.out.println();
System.out.println("Banking: RF=3 on all financial topics");
System.out.println("Rajesh's transfer survives two broker crashes");
}
}
Expected output:
RF=1: 1 failure = DATA LOST
RF=2: 1 failure = safe
RF=3: 2 failures = safe (banking standard)
How does ISR-based leader election provide availability?
When the leader's broker fails, the controller promotes a new leader from the ISR. Because every ISR member has all committed records, the new leader has the full committed history. Failover preserves data and resumes service quickly.
package com.bank.ft;
public class LeaderElectionFailover {
public static void main(String[] args) {
System.out.println("Leader Election Failover:");
System.out.println();
System.out.println("Before: ISR = {Broker1(L), Broker2, Broker3}");
System.out.println("Broker 1 crashes at 10:00:00");
System.out.println();
System.out.println("Controller detects failure");
System.out.println("Elects Broker 2 as new leader (ISR member)");
System.out.println("Broker 2 has offsets 0..1500 (all committed)");
System.out.println();
System.out.println("After: ISR = {Broker2(L), Broker3}");
System.out.println("Service resumed at 10:00:02 (2 seconds)");
System.out.println("No committed records lost");
System.out.println();
System.out.println("Rajesh's transfers: all safe, processing continues");
}
}
Expected output:
Broker 1 crashes -> Broker 2 elected -> 2 second failover
No committed records lost. Service continues.
How do acks and min.insync.replicas provide durability?
The acks setting controls when a producer considers a write successful. With acks=all, the leader acknowledges only after every ISR member has stored the record. min.insync.replicas sets the floor: if the ISR shrinks below this number, the leader rejects writes.
package com.bank.ft;
public class AcksAndMinIsr {
public static void main(String[] args) {
System.out.println("acks and min.insync.replicas:");
System.out.println();
System.out.println("acks=0: fire and forget (may lose data)");
System.out.println("acks=1: leader only (leader crash = lost)");
System.out.println("acks=all: all ISR members (strongest durability)");
System.out.println();
System.out.println("min.insync.replicas=2 with acks=all:");
System.out.println(" ISR={B1,B2,B3}: write -> B1,B2,B3 store -> ACK");
System.out.println(" ISR={B1,B2}: write -> B1,B2 store -> ACK");
System.out.println(" ISR={B1}: write -> REJECTED (ISR < min)");
System.out.println(" NotEnoughReplicasException thrown");
}
}
Expected output:
acks=all + min.insync.replicas=2:
3 ISR: write confirmed on 3 brokers
2 ISR: write confirmed on 2 brokers
1 ISR: write REJECTED (safety first)
How do all three mechanisms work together?
These three mechanisms must be configured together to actually get fault tolerance. Replication factor three alone is not enough if the producer uses acks=1, because a record acknowledged by only the leader can vanish if the leader dies before followers copy it.
package com.bank.ft;
public class ThreeMechanismsTogether {
public static void main(String[] args) {
System.out.println("All Three Mechanisms Together:");
System.out.println();
System.out.println("1. Replication Factor = 3");
System.out.println(" -> 3 copies on 3 brokers (redundancy)");
System.out.println();
System.out.println("2. ISR + Leader Election");
System.out.println(" -> New leader from caught-up replicas (availability)");
System.out.println();
System.out.println("3. acks=all + min.insync.replicas=2");
System.out.println(" -> Write confirmed on >= 2 brokers (durability)");
System.out.println();
System.out.println("Together: survive 1 broker failure with no data loss");
System.out.println(" and continued write availability");
}
}
Expected output:
Replication (redundancy) + ISR election (availability) + acks=all (durability)
= fault tolerance for banking
What are common mistakes with fault tolerance configuration?
The most common mistake is using acks=1 with replication factor three, thinking the data is safe because there are three copies. With acks=1, the leader acknowledges after storing locally, and a leader crash before followers copy the record loses it.
package com.bank.mistakes;
// WRONG: RF=3 but acks=1. Leader crash loses uncommitted records.
public class WrongAcksSetting {
public static void main(String[] args) {
System.out.println("WRONG: RF=3, acks=1");
System.out.println(" Leader stores record -> ACK sent to producer");
System.out.println(" Leader crashes BEFORE followers copy it");
System.out.println(" New leader elected -> record NOT there");
System.out.println(" Result: Rajesh's transfer LOST");
}
}
package com.bank.mistakes;
// CORRECT: RF=3, acks=all, min.insync.replicas=2
public class CorrectFaultTolerantConfig {
public static void main(String[] args) {
System.out.println("CORRECT: RF=3, acks=all, min.insync.replicas=2");
System.out.println(" Leader stores record");
System.out.println(" Followers store record");
System.out.println(" ACK sent to producer (on 2+ brokers)");
System.out.println(" Leader crashes -> new leader has the record");
System.out.println(" Result: Rajesh's transfer SAFE");
}
}
WRONG: RF=3 + acks=1 = false sense of security, data can be lost
CORRECT: RF=3 + acks=all + min.insync=2 = true fault tolerance
What failure scenarios should you plan for?
With the banking recipe (RF=3, min.insync=2, acks=all), one broker failure is fully tolerated: writes continue and no data is lost. If two brokers fail, the ISR drops below min.insync.replicas, so the leader stops accepting writes to protect durability. Reads may still work if one replica survives.
package com.bank.ft;
public class FailureScenarios {
public static void main(String[] args) {
System.out.println("Failure Scenarios (RF=3, min.insync=2, acks=all):");
System.out.println();
System.out.println("0 broker failures: normal operation");
System.out.println(" ISR=3, writes OK, reads OK");
System.out.println();
System.out.println("1 broker failure:");
System.out.println(" ISR=2 (>= min.insync=2), writes OK, reads OK");
System.out.println(" Leader elected from ISR, no data loss");
System.out.println();
System.out.println("2 broker failures:");
System.out.println(" ISR=1 (< min.insync=2), writes REJECTED");
System.out.println(" Reads may work if surviving replica is leader");
System.out.println(" Safety: no data loss, but service degraded");
System.out.println();
System.out.println("3 broker failures:");
System.out.println(" No replica available, partition OFFLINE");
}
}
Expected output:
0 failures: normal
1 failure: writes + reads OK, no data loss
2 failures: writes rejected, reads maybe OK
3 failures: partition offline
What should a senior engineer know?
Fault tolerance is not a single switch but a balance of durability, availability, and latency. For money movement you always choose durability, even at the cost of rejecting writes when too many brokers are down. Monitor under-replicated partitions and ISR-shrink alerts as early warnings that your safety margin is eroding.
package com.bank.senior;
public class FaultToleranceSeniorChecklist {
public static void main(String[] args) {
System.out.println("Senior Fault Tolerance Checklist:");
System.out.println("1. RF=3, min.insync=2, acks=all (banking recipe)");
System.out.println("2. unclean.leader.election.enable=false");
System.out.println("3. Rack-aware replica placement across zones");
System.out.println("4. Monitor: under-replicated partitions");
System.out.println("5. Monitor: ISR-shrink events");
System.out.println("6. Alert on min.insync.replicas threshold breach");
System.out.println("7. Plan for zone-level failures, not just broker");
System.out.println("8. Test failover regularly (chaos engineering)");
}
}
Expected output:
Senior checklist: banking recipe, rack-awareness,
monitoring ISR-shrink and under-replicated partitions,
testing failover regularly.
Visual Overview
Fault Tolerance: Three Mechanisms
==================================
Mechanism 1: Replication (Redundancy)
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Broker 1 │ │ Broker 2 │ │ Broker 3 │
│ Copy A │ │ Copy B │ │ Copy C │
└──────────┘ └──────────┘ └──────────┘
3 copies = survive 2 failures
Mechanism 2: ISR + Leader Election (Availability)
Broker 1 crashes:
ISR = {B2, B3} -> B2 elected leader
Service resumes in seconds, no data loss
Mechanism 3: acks=all + min.insync.replicas (Durability)
Producer -> B1 stores -> B2 stores -> ACK
(min.insync=2 met, write confirmed on 2 brokers)
Combined Result:
┌───────────────────────────────────────┐
│ 1 broker failure: │
│ Writes continue, reads continue │
│ No data lost │
│ │
│ 2 broker failures: │
│ Writes REJECTED (safety first) │
│ Reads may continue │
│ No data lost │
└───────────────────────────────────────┘
Key Takeaway
Fault tolerance in Kafka is achieved through three mechanisms working together: the replication factor for redundancy, ISR-based leader election for availability, and producer acks with min.insync.replicas for durability. No single mechanism is sufficient on its own. Replication factor three without acks=all can still lose data if the leader crashes before followers copy a record. acks=all without sufficient replication has no redundancy to protect.
The correct banking recipe is replication factor three, min.insync.replicas=2, acks=all, and unclean leader election disabled. With this configuration, a committed transfer always lives on at least two brokers. One broker failure is fully tolerated with continued writes and reads. Two broker failures cause writes to be rejected to protect durability, but no committed data is lost.
At National Bank of India, every money-movement topic uses exactly this configuration. A confirmed transfer survives a broker crash, and a leader failure triggers a seamless election from the ISR that keeps processing within seconds. The operations team monitors under-replicated partitions and ISR-shrink alerts as early warnings that the safety margin is eroding.
Remember one sentence: fault tolerance in Kafka requires replication for redundancy, ISR-based election for availability, and acks=all for durability, and all three must be configured together for the guarantee to hold.
---