Definition
- Replication : copying data from one database to one or more others to improve read scalability, availability, and durability.
- Master-slave (primary-replica) : a single primary accepts writes and replicates to read-only replicas that serve reads.
- Master-master : multiple nodes accept writes and must resolve conflicts when concurrent writes touch the same data.
- Replication lag : the delay before a write on the primary appears on a replica.
- Failover : promoting a replica to primary when the primary fails, requiring detection, promotion, and fencing.
National Bank of India keeps one official master ledger where all changes are written, and several photocopies that tellers across branches read from so they do not crowd the master. The copies lag the master by a tiny moment. If the master ledger is destroyed, one photocopy is promoted to become the new official master. If instead two branches could both write to their own copies, they might record conflicting changes to the same account, and the bank would need rules to reconcile them.
Understanding the Concepts
What is replication and why do we need it?
Replication copies data from one database to one or more others to improve read scalability, availability, and durability. Without it, a single database is both a performance bottleneck for reads and a single point of failure.
How does master-slave replication work?
Master-slave, also called primary-replica, has one node that accepts all writes and streams its change log to read-only replicas. Reads are spread across replicas to scale read-heavy workloads, while writes stay on the single primary.
package com.bank.replication;
public class ReadWriteRouter {
public void write(String sql) { primary.execute(sql); }
public String read(String sql) { return pickReplica().query(sql); }
}
What is replication lag and how do you handle it?
Replication lag is the delay before a write on the primary appears on a replica. Asynchronous replication acknowledges the write as soon as the primary commits, giving low write latency but allowing a replica to be briefly behind, causing read-your-own-write anomalies.
package com.bank.replication;
public class ReadAfterWrite {
public String read(String sql, boolean justWrote) {
// Route to primary if user just wrote, guaranteeing they see their change
return justWrote ? primary.query(sql) : pickReplica().query(sql);
}
}
What is master-master replication?
Master-master lets multiple nodes accept writes, improving write availability and regional locality. But it introduces conflicts when two nodes modify the same row concurrently. Conflicts must be resolved by last-write-wins (which can lose data) or application-specific merging. For money, master-master is dangerous because conflicting balance writes are unacceptable.
How does failover work?
Failover promotes a replica to primary when the primary fails. It must detect the failure, choose the most up-to-date replica, redirect writes to it, and fence off the old primary so it cannot resume and cause split-brain.
package com.bank.replication;
public class FailoverManager {
public void onPrimaryFailure() {
Replica best = replicas.stream()
.max(Comparator.comparingLong(Replica::lastAppliedLsn))
.orElseThrow();
best.promoteToLeader(); // new primary
oldPrimary.fence(); // prevent split-brain
updateDns(best.address()); // redirect writes
}
}
What is the difference between async and sync replication?
Asynchronous replication is fast but risks losing recently acknowledged writes during failover. Synchronous replication waits for at least one replica to acknowledge, trading latency for durability. Semi-synchronous is a middle ground.
| Mode | Write latency | Durability on failover | Use case |
|---|---|---|---|
| Async | Low | May lose recent writes | Social feeds |
| Semi-sync | Medium | At least one replica safe | General |
| Sync | High | All replicas safe | Banking ledger |
What is split-brain and how do you prevent it?
Split-brain occurs when two nodes both believe they are primary and accept writes, leading to divergent data. Prevention uses quorums (majority must agree on the leader) or fencing (STONITH : Shoot The Other Node In The Head) to ensure only one primary exists.
What are common replication topologies?
Single-leader with multiple replicas is the most common. Multi-leader is used for multi-region setups where each region has its own leader. Leaderless replication (like Cassandra) uses quorum reads and writes without a designated primary.
What should a senior engineer know?
A senior engineer resolves the async-vs-sync trade-off toward durability for banking. They configure connection draining during failover so in-flight transactions complete. They monitor replication lag as a key metric and alert when it exceeds a threshold, because growing lag means replicas are falling behind and read-your-write guarantees are weakening.
Visual Overview
Master-Slave Replication
┌──────────┐ write ┌──────────────┐
│ App │───────────►│ Primary │
│ Server │ │ (writes) │
└──────┬───┘ └──────┬───────┘
│ │ change log
│ read ┌─────┴─────┐
│ ┌────▼────┐ ┌────▼────┐
└───────────►│Replica 1│ │Replica 2│
│ (reads) │ │ (reads) │
└─────────┘ └─────────┘
lag: ~10ms lag: ~50ms
Failover:
┌──────────────┐ dies ┌─────────┐ promote ┌─────────────┐
│ Primary │────X───►│ Detect │──────────►│ Replica 1 │
│ (old) │ │ failure │ │ (new primary)│
└──────────────┘ └────┬────┘ └─────────────┘
│
fence old primary
(prevent split-brain)
Key Takeaway
Replication copies data from one database to others to improve read scalability, availability, and durability. The pattern I reach for first in banking is master-slave: a single primary accepts all writes and streams its change log to read-only replicas, and I spread reads across those replicas. This fits banking perfectly because the traffic is overwhelmingly reads.
The reality I always address is replication lag. With asynchronous replication the write is acknowledged as soon as the primary commits, which is fast but means a replica can be briefly behind, causing a read-your-own-writes anomaly. I handle this by routing a user's read to the primary right after they write, so Rajesh always sees his new balance immediately.
Master-master replication improves write availability but introduces conflicts when two nodes modify the same row. For money this is dangerous because conflicting balance writes are unacceptable, so a bank almost always keeps a single writer per piece of data.
Failover is the final piece: when the primary fails, promote the most up-to-date replica, redirect writes, and fence the old primary to prevent split-brain. The senior trade-off is that async replication risks losing recently acknowledged writes during failover, while sync replication protects them at higher latency. For banking I resolve toward durability, because losing an acknowledged transaction is far worse than a slightly slower write.
Remember one sentence: replicate for read scale and availability, route reads-after-writes to the primary to avoid staleness, and resolve the async-vs-sync trade-off toward durability for money.
---