Definition
- Replication factor – a per-topic setting that specifies the total number of copies (leader + followers) of each partition stored across different brokers.
- Fault tolerance – the number of broker failures a topic can survive, which equals the replication factor minus one.
min.insync.replicas– the minimum number of replicas that must acknowledge a write underacks=allbefore it is committed; works with the replication factor to control durability.- Rack-aware placement – configuring Kafka to spread replicas across different racks or availability zones so a zone-level failure does not lose all copies.
- Reassignment – the operational process of copying partition data to new brokers when increasing the replication factor of an existing topic.
Imagine National Bank of India decides how many backup copies of each ledger drawer to keep. A replication factor of three means every drawer is copied into three different branch vaults. If one vault is destroyed in a fire, two identical copies remain, so no records are lost and the bank keeps running. Choosing the number of copies is the replication factor: more copies means more safety, but also more storage and more work keeping them identical.
Understanding the Concepts
What is the replication factor?
The replication factor is set when you create a topic. It specifies how many copies of each partition Kafka maintains. A replication factor of three means each partition has three replicas on three separate brokers: one leader and two followers. Kafka places the replicas on different brokers deliberately, so a single broker failure never takes down more than one replica of any partition.
package com.bank.admin;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
import java.util.Collections;
import java.util.Properties;
public class ReplicationFactorExample {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
try (AdminClient admin = AdminClient.create(props)) {
// 6 partitions, replication factor 3
NewTopic topic = new NewTopic("transfers", 6, (short) 3);
admin.createTopics(Collections.singleton(topic)).all().get();
System.out.println("Created topic 'transfers': 6 partitions, RF=3");
System.out.println("Each partition has 3 copies on 3 brokers");
System.out.println("Tolerates up to 2 broker failures");
}
}
}
Expected output:
Created topic 'transfers': 6 partitions, RF=3
Each partition has 3 copies on 3 brokers
Tolerates up to 2 broker failures
How does the replication factor determine fault tolerance?
With a replication factor of N, the topic can tolerate up to N minus one broker failures and still have at least one surviving copy. A replication factor of one means no redundancy: if that broker dies, the partition's data is gone. A replication factor of three tolerates two broker failures.
package com.bank.replication;
public class FaultToleranceByRf {
public static void main(String[] args) {
for (int rf = 1; rf <= 5; rf++) {
int maxFailures = rf - 1;
System.out.println("RF=" + rf
+ " -> tolerates " + maxFailures + " broker failure(s)"
+ (rf == 1 ? " (NO redundancy!)" : ""));
}
}
}
Expected output:
RF=1 -> tolerates 0 broker failure(s) (NO redundancy!)
RF=2 -> tolerates 1 broker failure(s)
RF=3 -> tolerates 2 broker failure(s)
RF=4 -> tolerates 3 broker failure(s)
RF=5 -> tolerates 4 broker failure(s)
How does replication factor work with min.insync.replicas and acks?
The replication factor is the total number of copies, while min.insync.replicas is how many must acknowledge a write under acks=all. The typical safe combination is replication factor three and min.insync.replicas=2: every committed write lives on at least two brokers, and you can still accept writes after losing one broker.
| RF | min.insync.replicas | Broker failures tolerated (writes) | Broker failures tolerated (reads) |
|---|---|---|---|
| 1 | 1 | 0 | 0 |
| 2 | 1 | 1 | 1 |
| 3 | 2 | 1 | 2 |
| 3 | 3 | 0 | 2 |
| 5 | 3 | 2 | 4 |
package com.bank.config;
public class ReplicationConfig {
public static void main(String[] args) {
System.out.println("Banking Standard Configuration:");
System.out.println(" replication.factor = 3");
System.out.println(" min.insync.replicas = 2");
System.out.println(" acks = all");
System.out.println();
System.out.println("Guarantees:");
System.out.println(" Every confirmed write on >= 2 brokers");
System.out.println(" 1 broker failure: writes continue (ISR=2 >= min=2)");
System.out.println(" 2 broker failures: writes rejected (ISR=1 < min=2)");
System.out.println(" Reads still work with 1 surviving replica");
}
}
Expected output:
Banking Standard: RF=3, min.insync=2, acks=all
1 broker failure -> writes continue
2 broker failures -> writes rejected (safety over availability)
What are the use cases for different replication factors?
The replication factor is chosen based on how critical the data is. Financial topics use three for maximum safety. Non-critical telemetry may use one or two to save storage.
package com.bank.replication;
public class ReplicationFactorUseCases {
public static void main(String[] args) {
System.out.println("Replication Factor Use Cases:");
System.out.println();
System.out.println("RF=3 (banking standard):");
System.out.println(" - transfers: money movement (cannot lose)");
System.out.println(" - payments: bill payments (regulatory)");
System.out.println(" - audit-events: compliance logs (legal requirement)");
System.out.println(" - trade-events: stock trades (financial)");
System.out.println();
System.out.println("RF=2 (moderate importance):");
System.out.println(" - notification-events: customer alerts");
System.out.println(" - session-events: user sessions");
System.out.println();
System.out.println("RF=1 (low importance, dev/test):");
System.out.println(" - dev-metrics: regenerated easily");
System.out.println(" - test-events: throwaway data");
}
}
Expected output:
RF=3: transfers, payments, audit, trades (critical)
RF=2: notifications, sessions (moderate)
RF=1: dev metrics, test data (low, regeneratable)
What constraints does the replication factor have?
The replication factor cannot exceed the number of brokers in the cluster, because each replica must sit on a distinct broker. You can increase the replication factor of an existing topic, but it requires a reassignment that copies data to new brokers.
package com.bank.constraints;
public class ReplicationFactorConstraints {
public static void main(String[] args) {
System.out.println("Constraints:");
System.out.println(" RF <= number of brokers");
System.out.println(" 3-broker cluster: max RF = 3");
System.out.println(" 5-broker cluster: max RF = 5");
System.out.println();
System.out.println("Increasing RF on existing topic:");
System.out.println(" 1. Generate reassignment plan");
System.out.println(" 2. Execute reassignment (copies data over network)");
System.out.println(" 3. Verify completion");
System.out.println(" Impact: network I/O during copy, plan during low traffic");
}
}
Expected output:
RF <= number of brokers (hard constraint)
Increasing RF = data copy over network (plan during low traffic)
What is rack-aware replication?
Rack-aware placement spreads replicas across different racks or availability zones so a zone-level failure does not lose all copies. You configure broker.rack on each broker, and Kafka distributes replicas across racks.
package com.bank.replication;
public class RackAwareReplication {
public static void main(String[] args) {
System.out.println("Rack-Aware Replication (RF=3):");
System.out.println();
System.out.println("Broker 1: rack=zone-a -> Replica 1 (LEADER)");
System.out.println("Broker 2: rack=zone-b -> Replica 2 (FOLLOWER)");
System.out.println("Broker 3: rack=zone-c -> Replica 3 (FOLLOWER)");
System.out.println();
System.out.println("Zone-a goes down: Replicas in zone-b and zone-c survive");
System.out.println("No data lost, leader elected from surviving zone");
System.out.println();
System.out.println("Without rack-awareness:");
System.out.println(" All 3 replicas could be in zone-a");
System.out.println(" Zone-a failure = ALL data lost!");
}
}
Expected output:
Rack-aware: replicas in zone-a, zone-b, zone-c
Zone failure = 2 surviving replicas = no data loss
Without rack-awareness: all replicas could be in same zone
Zone failure = total data loss!
What are common mistakes with replication factor?
The most common mistake is using replication factor one in production for cost savings, not realizing that a single broker failure loses data permanently.
package com.bank.mistakes;
// WRONG: RF=1 for a financial topic in production.
public class WrongReplicationFactor {
public static void main(String[] args) {
System.out.println("WRONG: transfers topic with RF=1");
System.out.println(" Single broker failure = data permanently lost");
System.out.println(" No follower to take over = partition offline");
System.out.println(" Customer transfers GONE");
}
}
package com.bank.mistakes;
// CORRECT: RF=3 for all financial topics.
public class CorrectReplicationFactor {
public static void main(String[] args) {
System.out.println("CORRECT: transfers topic with RF=3");
System.out.println(" Two broker failures = data still survives");
System.out.println(" Follower promoted to leader = service continues");
System.out.println(" Customer transfers SAFE");
}
}
WRONG: RF=1 for financial data = one failure loses everything
CORRECT: RF=3 for financial data = survives two failures
What should a senior engineer know?
Each extra replica multiplies storage and inter-broker network traffic, because every write must be copied to every replica. Replication factor three roughly triples the storage and replication bandwidth compared to one. The senior decision is matching the replication factor to the data's importance.
package com.bank.senior;
public class ReplicationCostAnalysis {
public static void main(String[] args) {
long messagesPerDay = 10_000_000;
long avgMessageSize = 1024; // 1 KB
long dailyDataMb = (messagesPerDay * avgMessageSize) / (1024 * 1024);
System.out.println("Daily data per partition set: " + dailyDataMb + " MB");
System.out.println(" RF=1: " + dailyDataMb + " MB storage");
System.out.println(" RF=2: " + (dailyDataMb * 2) + " MB storage");
System.out.println(" RF=3: " + (dailyDataMb * 3) + " MB storage");
System.out.println();
System.out.println("Network: RF=3 means 2x replication traffic");
System.out.println("(each write copied to 2 followers)");
System.out.println();
System.out.println("Decision: RF=3 for money, RF=1 for throwaway data");
}
}
Expected output:
Daily data: ~9765 MB
RF=1: 9765 MB, RF=2: 19530 MB, RF=3: 29295 MB
Triple storage cost, but triple safety.
Visual Overview
Replication Factor = 3 ======================= Broker 1 (zone-a) Broker 2 (zone-b) Broker 3 (zone-c) ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ transfers-P0 (L) │ │ transfers-P0 (F) │ │ transfers-P0 (F) │ │ transfers-P1 (F) │ │ transfers-P1 (L) │ │ transfers-P1 (F) │ │ transfers-P2 (F) │ │ transfers-P2 (F) │ │ transfers-P2 (L) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ L = Leader, F = Follower Leaders spread across brokers for load distribution Rack-aware: each broker in different zone Broker 2 fails: ┌──────────────────┐ ┌──────────────────┐ │ transfers-P0 (L) │ │ transfers-P0 (F) │ │ transfers-P1 (F) │ Broker 3 elected │ transfers-P1 (L) │ │ transfers-P2 (F) │ leader for P1 ──────▶ │ transfers-P2 (L) │ └──────────────────┘ └──────────────────┘ Data survives. Service continues.
Key Takeaway
The replication factor is a per-topic setting that controls how many copies of each partition Kafka keeps across different brokers. A replication factor of three means every partition has three replicas on three separate brokers, tolerating up to two broker failures. It is the primary lever for durability and high availability, and the standard banking configuration pairs it with min.insync.replicas=2 and acks=all to ensure every committed write lives on at least two brokers.
The use cases come down to data importance. Financial topics like transfers, payments, and audit logs always use replication factor three so no committed record is ever lost. Non-critical, easily regenerated data like dashboard metrics may use a lower factor to save storage and network bandwidth. Rack-aware placement ensures replicas land in different availability zones, protecting against zone-level failures.
The main constraint is that the replication factor cannot exceed the number of brokers, and increasing it later requires data copying over the network. Each extra replica multiplies storage and replication traffic, so this is a cost-versus-safety decision that senior engineers make deliberately based on the data's business value.
Remember one sentence: the replication factor is the number of copies per partition, and it directly determines how many broker failures the topic can survive without losing data.
---