Back to Replication & Fault Tolerance
Q13Question 13 of 13

With min.insync.replicas=1, acks=all, replication factor 3: where is the actual durability risk?

Key Concepts
  • acks=all means all currently in-sync replicas, not all replicas; the ISR can shrink at runtime.
  • min.insync.replicas=1 lets the ISR shrink to just the leader, making acks=all acknowledge on one broker.
  • A single broker failure during this window loses already-acknowledged writes permanently.
  • The fix is min.insync.replicas=2 with replication factor 3 so acknowledged writes exist on at least two brokers.
  • With min.insync.replicas=2 and ISR too small, the producer gets NotEnoughReplicasException instead of silent loss.
  • Alert on under-minimum-ISR partitions because ISR=1 is a silent durability hole.

Definition

With `min.insync.replicas=1`, `acks=all` degrades to acknowledged by the leader alone whenever followers are out of sync, so a single broker failure can lose an already-acknowledged write -- set `min.insync.replicas=2` for real durability.
  • acks=all -- tells the producer to wait until all in-sync replicas (ISR) acknowledge, but "all" means all currently in-sync, not all replicas.
  • min.insync.replicas -- the floor for how small the ISR can be while still accepting writes; with 1, the leader alone satisfies the requirement.
  • The dangerous window -- when followers drop out of ISR (load, GC, restarts), the ISR shrinks to just the leader, and acks=all is satisfied by one broker.
  • Silent durability hole -- a message acknowledged while on one broker is lost forever if that broker's disk fails before a follower catches up.
  • The fix -- set min.insync.replicas=2 with replication factor 3 so every acknowledged write exists on at least two brokers.

Imagine National Bank of India promises that a transfer record is safe once three vaults each hold a copy. The policy says wait for all currently open vaults to confirm, but allows operating with as few as one open vault. When two vaults jam shut, the clerk records Rajesh's transfer in the one remaining vault, tells Rajesh it is safe in all open vaults, but all open vaults was just one. If that vault floods, Rajesh's transfer is gone, even though the bank promised it was safe.

Understanding the Concepts

What does acks=all actually promise?

It does not mean all replicas. It means all replicas that are currently in the in-sync replica set (ISR). A follower is in the ISR only while it keeps up with the leader within replica.lag.time.max.ms. The ISR can shrink at runtime.

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class AcksAllConfig {
    public static Properties create() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
        props.put(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));

        System.out.println("acks=all: wait for ALL in-sync replicas");
        System.out.println("  WARNING: 'all' means all CURRENTLY in ISR");
        System.out.println("  ISR can shrink to just the leader at runtime");
        return props;
    }

    public static void main(String[] args) {
        create();
    }
}
TEXT
Expected output:
acks=all: wait for ALL in-sync replicas
  WARNING: 'all' means all CURRENTLY in ISR
  ISR can shrink to just the leader at runtime

Where is the durability risk with min.insync.replicas=1?

The dangerous window: followers drop out of ISR (load, GC, restarts), ISR shrinks to just the leader, min.insync.replicas=1 is satisfied, and acks=all is satisfied by the leader alone. The producer gets a success for a message on one broker.

JAVA
package com.bank.kafka;

public class DurabilityRiskScenario {
    public static void main(String[] args) {
        System.out.println("Replication factor: 3, acks: all, min.insync.replicas: 1");
        System.out.println();
        System.out.println("Normal state:");
        System.out.println("  ISR = {broker-1 (leader), broker-2, broker-3}");
        System.out.println("  acks=all waits for all 3 -- safe");
        System.out.println();
        System.out.println("Dangerous window:");
        System.out.println("  Broker-2 GC pause, broker-3 restarting");
        System.out.println("  ISR = {broker-1 (leader)} -- just ONE");
        System.out.println("  min.insync.replicas=1 is satisfied");
        System.out.println("  acks=all satisfied by leader alone");
        System.out.println("  Producer gets SUCCESS for Rajesh's transfer");
        System.out.println();
        System.out.println("Disaster:");
        System.out.println("  Broker-1 disk fails before followers catch up");
        System.out.println("  Rajesh's transfer is GONE FOREVER");
        System.out.println("  Producer already reported SUCCESS");
    }
}
TEXT
Expected output:
Replication factor: 3, acks: all, min.insync.replicas: 1

Normal state:
  ISR = {broker-1 (leader), broker-2, broker-3}
  acks=all waits for all 3 -- safe

Dangerous window:
  Broker-2 GC pause, broker-3 restarting
  ISR = {broker-1 (leader)} -- just ONE
  min.insync.replicas=1 is satisfied
  acks=all satisfied by leader alone
  Producer gets SUCCESS for Rajesh's transfer

Disaster:
  Broker-1 disk fails before followers catch up
  Rajesh's transfer is GONE FOREVER
  Producer already reported SUCCESS

How does min.insync.replicas=2 fix this?

With replication factor 3 and min.insync.replicas=2, the leader refuses any acks=all write unless at least two replicas are in-sync. Every acknowledged message exists on at least two brokers.

BASH
# Set min.insync.replicas=2 on the settlements topic
kafka-topics.sh --alter --topic settlements \
  --config min.insync.replicas=2 \
  --bootstrap-server kafka-1:9092

# Verify
kafka-configs.sh --describe --topic settlements \
  --bootstrap-server kafka-1:9092
TEXT
Dynamic configs for topic settlements:
  min.insync.replicas=2

What happens when a second replica is unavailable?

With min.insync.replicas=2, if only one replica is in-sync, the producer gets NotEnoughReplicasException and must retry. You trade a little availability for durability.

JAVA
package com.bank.kafka;

public class NotEnoughReplicasDemo {
    public static void main(String[] args) {
        System.out.println("min.insync.replicas=2, ISR has only 1 replica:");
        System.out.println("  Producer gets: NotEnoughReplicasException");
        System.out.println("  Write REJECTED -- better than silent data loss");
        System.out.println("  Producer retries until ISR recovers to >= 2");
        System.out.println();
        System.out.println("Trade-off:");
        System.out.println("  min.insync.replicas=1: always accepts writes, risks loss");
        System.out.println("  min.insync.replicas=2: may reject writes, guarantees durability");
    }
}
TEXT
Expected output:
min.insync.replicas=2, ISR has only 1 replica:
  Producer gets: NotEnoughReplicasException
  Write REJECTED -- better than silent data loss
  Producer retries until ISR recovers to >= 2

Trade-off:
  min.insync.replicas=1: always accepts writes, risks loss
  min.insync.replicas=2: may reject writes, guarantees durability

How do the three settings form a durability contract?

Durability requires all three settings to agree. Miss any one and you have a silent hole.

| Setting | Purpose | Unsafe Value | Safe Value |
|---|---|---|---|
| Replication factor | Number of copies | 1 | 3 |
| acks | When producer gets success | 0 or 1 | all |
| min.insync.replicas | Floor for ISR size | 1 | 2 |

JAVA
package com.bank.kafka;

public class DurabilityContract {
    public static void main(String[] args) {
        System.out.println("Durability contract -- all three must agree:");
        System.out.println("  Replication factor = 3 (gives you the copies)");
        System.out.println("  acks = all (makes producer wait for ISR)");
        System.out.println("  min.insync.replicas = 2 (ISR is never just 1 broker)");
        System.out.println();
        System.out.println("Result: survives loss of any single broker");
        System.out.println("  without losing acknowledged data");
    }
}
TEXT
Expected output:
Durability contract -- all three must agree:
  Replication factor = 3 (gives you the copies)
  acks = all (makes producer wait for ISR)
  min.insync.replicas = 2 (ISR is never just 1 broker)

Result: survives loss of any single broker
  without losing acknowledged data

What about unclean.leader.election.enable?

If unclean.leader.election.enable=true, a broker not in the ISR can be elected leader after a failure, which can lose acknowledged data. For banking, keep it false.

BASH
# Disable unclean leader election (should be the default)
kafka-configs.sh --alter --entity-type brokers --entity-default \
  --add-config unclean.leader.election.enable=false \
  --bootstrap-server kafka-1:9092

What metrics should you monitor?

Alert on under-replicated and at-minimum-ISR partitions. A partition at ISR=1 is a silent durability hole.

BASH
# Check for under-replicated partitions
kafka-topics.sh --describe --under-replicated-partitions \
  --bootstrap-server kafka-1:9092

# Monitor ISR count per partition
kafka-topics.sh --describe --topic settlements --bootstrap-server kafka-1:9092

What should a senior engineer know?

Even RF=3 with min.insync.replicas=2 is not immune to correlated failure (losing two brokers in the same rack). Spread replicas across racks or zones using rack awareness. Alert on the under-minimum-ISR metric because a partition at ISR=1 looks healthy on a dashboard right up until it loses data.

Visual Overview

DIAGRAM
 Durability Risk with min.insync.replicas=1
 ============================================

 Normal state (ISR = 3):
 ┌──────────┐  ┌──────────┐  ┌──────────┐
 │ Broker 1 │  │ Broker 2 │  │ Broker 3 │
 │ (Leader) │  │ (ISR)    │  │ (ISR)    │
 │ ✓ data   │  │ ✓ data   │  │ ✓ data   │
 └──────────┘  └──────────┘  └──────────┘
  acks=all waits for 3 → SAFE


 Dangerous window (ISR = 1):
 ┌──────────┐  ┌──────────┐  ┌──────────┐
 │ Broker 1 │  │ Broker 2 │  │ Broker 3 │
 │ (Leader) │  │ (OUT)    │  │ (OUT)    │
 │ ✓ data   │  │ GC pause │  │ restart  │
 └──────────┘  └──────────┘  └──────────┘
  min.insync.replicas=1 → satisfied
  acks=all waits for 1 → acknowledged
  Message on ONE broker only!


 Fix: min.insync.replicas=2
 ┌──────────┐  ┌──────────┐  ┌──────────┐
 │ Broker 1 │  │ Broker 2 │  │ Broker 3 │
 │ (Leader) │  │ (OUT)    │  │ (OUT)    │
 └──────────┘  └──────────┘  └──────────┘
  ISR = 1 < min.insync.replicas = 2
  Producer gets: NotEnoughReplicasException
  Write REJECTED → better than silent loss

Key Takeaway

The setting acks=all means all replicas currently in the in-sync set, not all replicas. And min.insync.replicas is the floor for how small that set can be. With min.insync.replicas=1, the ISR can shrink to just the leader, and acks=all is satisfied by one broker. If that broker's disk fails, the acknowledged message is gone.

The fix is min.insync.replicas=2 with replication factor 3. Now the leader refuses writes unless at least two replicas are in-sync, so every acknowledged message survives any single broker failure.

In my National Bank of India settlement service, a topic with min.insync.replicas=1 lost an acknowledged settlement when the ISR shrank to one broker during a GC pause and that broker's disk failed. We changed to min.insync.replicas=2, kept acks=all, and disabled unclean leader election.

Remember one sentence: durability needs all three settings -- replication factor 3, acks=all, and min.insync.replicas=2 -- because min.insync.replicas=1 lets acks=all degrade to one broker.

---

Share
Use ← → arrow keys to navigate