Definition
- Primary (master) – The single node that accepts all writes and is the authoritative source of truth.
- Replica (slave) – One or more nodes that receive a continuous copy of the primary's data and serve read traffic.
- Async replication – Primary commits and acknowledges immediately, ships changes afterward; fast but risks losing recent writes on primary failure.
- Sync replication – Primary waits for at least one replica to confirm before acknowledging; safe but adds write latency.
- Failover – Promoting a replica to primary when the old primary fails; can be manual or automated.
Imagine National Bank of India keeps one official master ledger that only the head cashier can write to. Several assistants keep photocopies and serve customers who just want to read their balance. If one assistant is off sick, customers ask another. But if the head cashier suddenly leaves, nobody can record new transactions until one assistant is promoted, and any entry the cashier had written but not yet shared could be lost.
Understanding the Concepts
What is primary-replica architecture?
All writes go to one node (primary). Replicas receive a continuous stream of changes and serve reads.
package com.bank.replication;
public class PrimaryReplicaConcept {
public static void main(String[] args) {
System.out.println("Primary-Replica architecture:");
System.out.println(" Primary: accepts ALL writes (INSERT, UPDATE, DELETE)");
System.out.println(" Replica 1: receives changes, serves reads");
System.out.println(" Replica 2: receives changes, serves reads");
System.out.println();
System.out.println("Write path: Client -> Primary -> commit -> replicate to replicas");
System.out.println("Read path: Client -> any Replica (or Primary)");
}
}
Expected output:
Primary-Replica architecture:
Primary: accepts ALL writes (INSERT, UPDATE, DELETE)
Replica 1: receives changes, serves reads
Replica 2: receives changes, serves reads
Write path: Client -> Primary -> commit -> replicate to replicas
Read path: Client -> any Replica (or Primary)
How does async vs sync replication work?
Async is fast but risks data loss. Sync is safe but adds latency. Semi-sync is the common middle ground.
package com.bank.replication;
public class ReplicationModes {
public static void main(String[] args) {
System.out.println("Async replication:");
System.out.println(" Primary commits -> ACK to client immediately");
System.out.println(" Ships changes to replicas afterward");
System.out.println(" Fast, but if primary dies before shipping -> data lost");
System.out.println();
System.out.println("Sync replication:");
System.out.println(" Primary commits -> waits for replica ACK -> then ACK to client");
System.out.println(" Safe, but every write is slower by network round-trip");
System.out.println();
System.out.println("Semi-sync (common):");
System.out.println(" Primary waits for ONE replica ACK -> ACK to client");
System.out.println(" Balance: one replica guaranteed, minimal latency");
}
}
Expected output:
Async replication:
Primary commits -> ACK to client immediately
Ships changes to replicas afterward
Fast, but if primary dies before shipping -> data lost
Sync replication:
Primary commits -> waits for replica ACK -> then ACK to client
Safe, but every write is slower by network round-trip
Semi-sync (common):
Primary waits for ONE replica ACK -> ACK to client
Balance: one replica guaranteed, minimal latency
What happens when a replica dies?
Impact is limited: read traffic shifts to other replicas. No data loss. Writes unaffected.
package com.bank.replication;
public class ReplicaFailure {
public static void main(String[] args) {
System.out.println("Replica failure:");
System.out.println(" Replica-2 crashes");
System.out.println(" Read traffic shifts to Replica-1 and Primary");
System.out.println(" No data loss (Primary still has everything)");
System.out.println(" Writes completely unaffected");
System.out.println(" When Replica-2 recovers: reconnects and catches up from log");
}
}
Expected output:
Replica failure:
Replica-2 crashes
Read traffic shifts to Replica-1 and Primary
No data loss (Primary still has everything)
Writes completely unaffected
When Replica-2 recovers: reconnects and catches up from log
What happens when the primary dies?
All writes stop. Recovery means promoting a replica. Risks: data loss and split-brain.
package com.bank.replication;
public class PrimaryFailure {
public static void main(String[] args) {
System.out.println("Primary failure:");
System.out.println(" All writes STOP immediately");
System.out.println(" Reads from replicas continue but grow stale");
System.out.println();
System.out.println("Recovery: promote most up-to-date replica to new primary");
System.out.println(" Manual: DBA runs promotion commands");
System.out.println(" Automated: failover controller detects, selects, promotes");
System.out.println();
System.out.println("Risks:");
System.out.println(" 1. Data loss: writes primary committed but didn't replicate");
System.out.println(" 2. Split-brain: network partition -> two nodes both act as primary");
System.out.println(" Fix: quorum + fencing (only one primary can exist)");
}
}
Expected output:
Primary failure:
All writes STOP immediately
Reads from replicas continue but grow stale
Recovery: promote most up-to-date replica to new primary
Manual: DBA runs promotion commands
Automated: failover controller detects, selects, promotes
Risks:
1. Data loss: writes primary committed but didn't replicate
2. Split-brain: network partition -> two nodes both act as primary
What about replication lag?
A client who writes and immediately reads from a replica may not see their own write.
package com.bank.replication;
import java.math.BigDecimal;
public class ReplicationLag {
public static void main(String[] args) {
System.out.println("Replication lag scenario:");
System.out.println(" Rajesh transfers 50000 (write to Primary)");
System.out.println(" Rajesh views balance (read from Replica)");
System.out.println(" Replica has not received the transfer yet");
System.out.println(" Rajesh sees OLD balance -> confusing!");
System.out.println();
System.out.println("Mitigations:");
System.out.println(" 1. Read-your-own-writes: route to Primary briefly after write");
System.out.println(" 2. Track replication position: only read caught-up replicas");
System.out.println(" 3. Return write result directly: 'New balance: 950000'");
}
}
Expected output:
Replication lag scenario:
Rajesh transfers 50000 (write to Primary)
Rajesh views balance (read from Replica)
Replica has not received the transfer yet
Rajesh sees OLD balance -> confusing!
What are the use cases?
Read scaling, high availability, backups, analytics offloading, and geographic locality.
| Use Case | How It Works |
|---|---|
| Read scaling | Offload heavy reads to replicas, free the primary for writes |
| High availability | Replicas hold full copies; failover keeps the system running |
| Backups | Take backups from a replica without slowing the primary |
| Analytics | Run expensive reports against a replica |
| Geo locality | Place replicas near users for lower read latency |
package com.bank.replication;
public class UseCases {
public static void main(String[] args) {
System.out.println("At National Bank of India:");
System.out.println(" Balance lookups (90% of traffic) -> Replica");
System.out.println(" Transfers (10% of traffic) -> Primary");
System.out.println(" Nightly reports -> Replica (no impact on Primary)");
System.out.println(" Primary fails -> Replica promoted in 30 seconds");
}
}
Expected output:
At National Bank of India:
Balance lookups (90% of traffic) -> Replica
Transfers (10% of traffic) -> Primary
Nightly reports -> Replica (no impact on Primary)
Primary fails -> Replica promoted in 30 seconds
What is the limitation?
Primary-replica scales reads but not writes. All writes still go to one node. For write scalability, you need sharding.
package com.bank.replication;
public class Limitation {
public static void main(String[] args) {
System.out.println("Limitation: does NOT scale writes");
System.out.println(" 10 replicas -> 10x read capacity");
System.out.println(" But still 1 primary -> same write capacity");
System.out.println();
System.out.println("For write scalability:");
System.out.println(" -> Shard the database (split rows across multiple primaries)");
System.out.println(" -> Each shard has its own primary-replica pair");
}
}
Expected output:
Limitation: does NOT scale writes
10 replicas -> 10x read capacity
But still 1 primary -> same write capacity
For write scalability:
-> Shard the database (split rows across multiple primaries)
-> Each shard has its own primary-replica pair
What should a senior engineer know?
Monitor replication lag as a first-class metric. Use semi-sync for critical data. Guard against split-brain with fencing and quorum.
Senior checklist:
[x] Monitor replication lag (Prometheus gauge per replica)
[x] Semi-sync replication for critical banking data
[x] Automated failover with quorum and fencing
[x] Route read-your-own-writes to Primary
[x] Old primary rejoins as replica after recovery
[x] Understand: scales reads, NOT writes
[x] For write scale: shard each shard with its own primary-replica pair
Visual Overview
Primary-Replica Architecture:
Write Path:
Client ──▶ Primary ──▶ Commit ──▶ Replicate
│
┌───────────┼───────────┐
▼ ▼ ▼
Replica-1 Replica-2 Replica-3
Read Path:
Client ──▶ Load Balancer ──▶ Replica-1 (or 2 or 3)
Failure Scenarios:
Replica dies: Read traffic shifts. No data loss. Writes unaffected.
Primary dies: Writes STOP. Promote replica. Risk: data loss + split-brain.
Failover:
┌─────────┐ dead ┌──────────────┐ promote ┌───────────────┐
│ Primary │───────▶│ Failover │──────────▶│ Replica-1 │
│ (dead) │ │ Controller │ │ (new Primary) │
└─────────┘ │ (detects, │ └───────────────┘
│ selects, │
│ fences old) │
└──────────────┘
Key Takeaway
Primary-replica is a replication architecture with one authoritative writer and multiple read copies. All writes go to the primary, which records them in a replication log. Replicas receive the stream of changes and serve read traffic. This scales reads, provides high availability, and enables backups and analytics without burdening the primary.
The failure scenarios are asymmetric. A dead replica is graceful: reads shift to others, no data loss, writes unaffected, and the repaired replica catches up. A dead primary is serious: writes stop, reads grow stale, and recovery means promoting the most up-to-date replica. The risks are data loss (writes committed but not yet replicated) and split-brain (two nodes both acting as primary), which robust failover prevents with quorum and fencing.
At National Bank of India, Pankaj moved all reads to replicas to relieve an overloaded database. When a replica crashed, there was no impact. When the primary failed, they promoted a replica within 30 seconds, losing only the last few uncommitted writes. For critical data they switched to semi-sync replication so an acknowledged write was guaranteed to survive a primary failure. The key limitation is that primary-replica scales reads but not writes.
Remember one sentence: one primary takes all writes and replicates to replicas for read scaling and availability; a dead replica just loses read capacity, while a dead primary halts writes until promoted, risking loss of not-yet-replicated writes.
---