Definition
- Redundant pair -- two load balancer nodes share a floating virtual IP (VIP); the active node serves and the passive takes over on failure.
- VRRP/keepalived -- a protocol that manages the VIP: nodes exchange heartbeats and the passive takes the VIP if the active stops.
- Active-active -- both nodes serve traffic simultaneously; uses DNS or upstream distribution.
- DNS failover -- health checks stop returning an unreachable zone's address; bounded by TTL caching.
- Managed cloud LBs -- internally redundant, multi-AZ services; not a single box.
- Test failover -- an untested standby often fails when needed.
The floor manager at National Bank of India directs every customer, but if there is only one manager and they faint, the branch jams. The fix: two managers sharing the same badge (VIP). They check on each other constantly, and the instant the active one faints, the standby grabs the badge. At the city level, if one whole branch is unreachable, customers are sent to another (DNS failover).
Understanding the Concepts
Why is a single load balancer a SPOF?
All traffic flows through the load balancer. If it dies, nothing reaches the healthy backends. You moved the single point of failure from the backends to the balancer.
package com.bank.infra;
public class SingleLbSpof {
public static void main(String[] args) {
int healthyBackends = 10;
boolean lbAlive = false;
if (!lbAlive) {
System.out.println(healthyBackends + " backends are healthy");
System.out.println("But the single LB is dead");
System.out.println("Result: 0 requests reach any backend");
System.out.println("You moved the SPOF from backends to the LB");
}
}
}
Expected output:
10 backends are healthy
But the single LB is dead
Result: 0 requests reach any backend
You moved the SPOF from backends to the LB
How does a redundant pair with VIP work?
Two LB nodes run, and the VIP (Virtual IP) is assigned to whichever is active. VRRP via keepalived manages heartbeats. If the active stops sending heartbeats, the passive takes the VIP within seconds.
package com.bank.infra;
public class VipFailover {
public static void main(String[] args) {
String vip = "10.0.1.100";
String activeLb = "LB-A";
String passiveLb = "LB-B";
System.out.println("VIP " + vip + " owned by " + activeLb);
System.out.println(activeLb + " stops sending heartbeats...");
System.out.println(passiveLb + " detects failure via VRRP");
System.out.println(passiveLb + " takes VIP " + vip + " in < 3 seconds");
System.out.println("Clients see no change: same IP, new node");
}
}
Expected output:
VIP 10.0.1.100 owned by LB-A
LB-A stops sending heartbeats...
LB-B detects failure via VRRP
LB-B takes VIP 10.0.1.100 in < 3 seconds
Clients see no change: same IP, new node
What is active-active load balancing?
Both nodes serve traffic simultaneously. DNS or an upstream layer distributes across both VIPs. Better capacity utilization but more complex: both must handle their share, and you need a mechanism to redistribute if one dies.
package com.bank.infra;
public class ActiveActive {
public static void main(String[] args) {
System.out.println("DNS returns two addresses:");
System.out.println(" LB-A: 10.0.1.100 (Zone-A)");
System.out.println(" LB-B: 10.0.2.100 (Zone-B)");
System.out.println("Traffic split across both");
System.out.println("If LB-A dies: DNS health check removes it");
System.out.println("All traffic flows to LB-B until LB-A recovers");
}
}
Expected output:
DNS returns two addresses:
LB-A: 10.0.1.100 (Zone-A)
LB-B: 10.0.2.100 (Zone-B)
Traffic split across both
If LB-A dies: DNS health check removes it
All traffic flows to LB-B until LB-A recovers
How does DNS failover work?
Run load balancers in multiple availability zones. DNS health checks (e.g. Route 53) stop returning a dead zone's address. The caveat: DNS is cached for TTL, so DNS failover is slower than VIP failover. Keep TTLs low (e.g. 60 seconds) for failover-critical records.
package com.bank.infra;
public class DnsFailover {
public static void main(String[] args) {
int ttlSeconds = 60;
System.out.println("Route 53 health check: ping LB every 10s");
System.out.println("LB Zone-B fails health check 3 times");
System.out.println("Route 53 stops returning Zone-B address");
System.out.println("Clients with cached DNS: up to " + ttlSeconds + "s stale");
System.out.println("After TTL expires: all traffic to Zone-A");
System.out.println("Tip: keep TTL low (" + ttlSeconds + "s) for failover records");
}
}
Expected output:
Route 53 health check: ping LB every 10s
LB Zone-B fails health check 3 times
Route 53 stops returning Zone-B address
Clients with cached DNS: up to 60s stale
After TTL expires: all traffic to Zone-A
Tip: keep TTL low (60s) for failover records
What about managed cloud load balancers?
AWS ALB/NLB, GCP, and Azure LBs are not a single box. They are internally redundant, horizontally scaled, multi-AZ services. For region-level resilience, place LBs in multiple regions with DNS or global load balancing.
package com.bank.infra;
public class ManagedLb {
public static void main(String[] args) {
System.out.println("AWS ALB: internally redundant, spans multiple AZs");
System.out.println("Not a single box; AWS manages the fleet");
System.out.println("For region failure: deploy in us-east-1 + eu-west-1");
System.out.println("Route 53 health-checked DNS routes between regions");
}
}
Expected output:
AWS ALB: internally redundant, spans multiple AZs
Not a single box; AWS manages the fleet
For region failure: deploy in us-east-1 + eu-west-1
Route 53 health-checked DNS routes between regions
How does state on the load balancer affect failover?
Sticky-session tables complicate failover: when the active LB dies, its session table is lost and users are redistributed, potentially losing session affinity. Prefer stateless load balancing or externalized session state (Redis).
package com.bank.infra;
public class StatefulVsStateless {
public static void main(String[] args) {
// WRONG: sticky sessions stored on LB
System.out.println("WRONG: LB-A stores sticky-session table");
System.out.println("LB-A dies --> session table lost --> users reassigned");
// CORRECT: stateless LB, sessions in Redis
System.out.println("CORRECT: LB is stateless, sessions in Redis");
System.out.println("LB failover is seamless: no session state to lose");
}
}
Expected output:
WRONG: LB-A stores sticky-session table
LB-A dies --> session table lost --> users reassigned
CORRECT: LB is stateless, sessions in Redis
LB failover is seamless: no session state to lose
What are common mistakes?
The most common mistake is never testing the failover. An untested standby often fails when needed.
package com.bank.infra;
public class CommonMistakes {
public static void main(String[] args) {
// WRONG: assuming managed LB in one AZ is enough
System.out.println("WRONG: managed LB in one AZ -- zone outage takes it down");
System.out.println("CORRECT: configure across multiple AZs");
// WRONG: never testing failover
System.out.println("WRONG: standby LB never tested -- fails when needed");
System.out.println("CORRECT: chaos test -- kill active LB, verify traffic shifts");
}
}
Expected output:
WRONG: managed LB in one AZ -- zone outage takes it down
CORRECT: configure across multiple AZs
WRONG: standby LB never tested -- fails when needed
CORRECT: chaos test -- kill active LB, verify traffic shifts
What should a senior engineer know?
- VIP failover gives sub-second recovery inside a zone.
- DNS failover covers zone and region outages but is bounded by TTL caching.
- Managed multi-AZ LBs handle most of this internally.
- In cloud environments, raw VRRP may not work (no L2 broadcast); lean on managed LBs and DNS.
- Always verify the failover works by testing it under realistic conditions.
package com.bank.infra;
public class SeniorInsights {
public static void main(String[] args) {
System.out.println("1. VIP failover: sub-second, in-zone");
System.out.println("2. DNS failover: covers zones/regions, bounded by TTL");
System.out.println("3. Managed LBs: internally redundant, multi-AZ");
System.out.println("4. Cloud: VRRP may not work; use managed + DNS");
System.out.println("5. Always test failover under realistic load");
}
}
Expected output:
1. VIP failover: sub-second, in-zone
2. DNS failover: covers zones/regions, bounded by TTL
3. Managed LBs: internally redundant, multi-AZ
4. Cloud: VRRP may not work; use managed + DNS
5. Always test failover under realistic load
Visual Overview
Load Balancer Redundancy Stack
Region Level:
┌──────────────────────────────────────────┐
│ DNS (Route 53) health-checked failover │
└──────────┬───────────────┬───────────────┘
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Region-US │ │ Region-EU │
└───────┬────────┘ └───────┬────────┘
▼ ▼
Zone Level:
┌────────────────┐ ┌────────────────┐
│ LB Active/VIP │ │ LB Standby │
│ VRRP heartbeat │◄►│ takes VIP on │
│ │ │ failure │
└───────┬────────┘ └────────────────┘
▼
┌────────────────┐
│ Backends │
└────────────────┘
Active-Passive VIP Failover
┌──────────────┐ ┌──────────────┐
│ LB-Active │◄─────►│ LB-Passive │
│ owns VIP │ hbeat │ standby │
└──────┬───────┘ └──────────────┘
│
VIP: 10.0.1.100
│
┌──────▼───────┐
│ Backends │
└──────────────┘
If Active dies --> Passive takes VIP in seconds
Key Takeaway
The irony of load balancing is that if you are not careful you simply move the single point of failure from the backends to the load balancer. All traffic flows through it, so if it dies the whole system is unreachable even though every backend is healthy.
The primary fix is a redundant pair sharing a floating VIP with heartbeat failover. VRRP via keepalived has the nodes exchange heartbeats, and when the active stops the passive takes the VIP within seconds. DNS failover covers zone and region outages: health checks stop returning a dead zone's address. Managed cloud LBs are internally redundant and span multiple AZs, handling most of this automatically.
At National Bank of India, Pankaj ran an active-passive pair with a VIP for sub-second in-zone failover, and Route 53 health-checked DNS for multi-zone coverage. The most important lesson was to always test the failover, because an untested standby often fails when needed.
Remember one sentence: never run a single load balancer; run a redundant pair with a floating VIP for fast in-zone failover, add DNS failover for zone outages, lean on managed multi-AZ load balancers in the cloud, and always test the failover.
---