Definition
- Redundancy across failure domains – Run enough instances spread across multiple nodes and availability zones so a single failure never takes all replicas.
- Kubernetes self-healing – Declare a desired replica count; the controller continuously drives the actual count back to that number, recreating crashed pods automatically.
- Health probes – Liveness probes trigger a restart of unhealthy pods; readiness probes gate traffic so cold or warming pods do not receive requests.
- Auto-scaling – The Horizontal Pod Autoscaler watches CPU or custom metrics and adds pods when survivors are overloaded.
- PodDisruptionBudget – Tells Kubernetes the minimum pods that must stay available during voluntary disruptions like node upgrades.
Imagine the National Bank of India runs a critical processing room with two clerks. One morning both clerks fall sick and work grinds to a halt. A fault-tolerant bank never relies on just two clerks in one room. It keeps clerks in several rooms across several buildings, a supervisor who checks each clerk constantly and calls in replacements when one stops, and a staffing policy that pulls in extra clerks when the queue grows. Now if two clerks drop, the others absorb the work and fresh clerks arrive within minutes.
Understanding the Concepts
Why is running two instances on the same node not enough?
Running two instances on the same node gives almost no protection because a node hardware failure, kernel panic, or power loss kills both at once. Replicas must be spread across independent failure domains.
# WRONG: Both replicas can land on the same node
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
spec:
replicas: 2
# No anti-affinity -> scheduler may place both on node-1
# CORRECT: Anti-affinity forces replicas onto different nodes
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
spec:
replicas: 3
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: payment-service
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: payment-service
Result:
pod-1 -> node-a (zone ap-south-1a)
pod-2 -> node-b (zone ap-south-1b)
pod-3 -> node-c (zone ap-south-1c)
A whole-zone outage still leaves 2 healthy pods.
How does Kubernetes self-healing work?
You declare a desired replica count in a Deployment. The ReplicaSet controller continuously drives the actual running count back to that declaration. If two pods crash, it immediately creates two new ones.
package com.bank.k8s;
public class SelfHealingDemo {
public static void main(String[] args) {
System.out.println("Declared replicas: 3");
System.out.println("Pod-1 crashes -> actual = 2 -> controller creates pod-4");
System.out.println("Pod-2 crashes -> actual = 2 -> controller creates pod-5");
System.out.println("Desired state restored: 3 running pods");
System.out.println("No human intervention required.");
}
}
Expected output:
Declared replicas: 3
Pod-1 crashes -> actual = 2 -> controller creates pod-4
Pod-2 crashes -> actual = 2 -> controller creates pod-5
Desired state restored: 3 running pods
No human intervention required.
How do liveness and readiness probes work?
A liveness probe answers whether the process should be killed and restarted. A readiness probe answers whether the pod should receive traffic. A pod that is alive but still warming its connection pool should report not-ready.
package com.bank.health;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
import java.util.concurrent.atomic.AtomicBoolean;
@Component
public class ReadinessHealthIndicator implements HealthIndicator {
private final AtomicBoolean cacheWarmed = new AtomicBoolean(false);
public void markReady() {
cacheWarmed.set(true);
System.out.println("Cache warmed. Service is ready for traffic.");
}
@Override
public Health health() {
if (cacheWarmed.get()) {
return Health.up().withDetail("cache", "warmed").build();
}
return Health.down().withDetail("cache", "still warming").build();
}
}
# Kubernetes probe configuration
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
failureThreshold: 3
periodSeconds: 10
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
failureThreshold: 3
periodSeconds: 5
Pod startup sequence:
t=0s: Pod created, liveness=UP, readiness=DOWN (cache warming)
t=15s: Cache warmed, readiness=UP -> traffic starts flowing
t=60s: Database hangs, liveness fails 3x -> pod killed and restarted
How does auto-scaling restore and grow capacity?
The Horizontal Pod Autoscaler watches metrics and changes the replica count to match demand. If a crash leaves survivors overloaded, the HPA adds pods.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: payment-service-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: payment-service
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
Scenario: 2 of 3 pods crash during month-end salary processing
Survivor CPU jumps to 95% -> HPA scales to 5 replicas
New pods pass readiness -> traffic spreads across 5
Load stabilizes at 60% CPU per pod
What is a PodDisruptionBudget and why does it matter?
A PDB tells Kubernetes the minimum pods that must stay available during voluntary disruptions like node drains and upgrades, preventing maintenance from accidentally taking you to zero.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: payment-service-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: payment-service
With minAvailable: 2 and 3 replicas:
Node drain on node-a -> Kubernetes checks: will draining leave >= 2 pods?
Yes (2 remain on node-b, node-c) -> drain proceeds
If only 2 were running -> drain is BLOCKED until a new pod is scheduled
Why must the data layer also be fault tolerant?
Resilient application instances in front of a single-point-of-failure database is a false sense of security. The database needs its own replication and failover.
package com.bank.data;
public class DataLayerFaultTolerance {
public static void main(String[] args) {
System.out.println("Application tier: 3 pods across 3 zones -> fault tolerant");
System.out.println("Database tier: single instance -> SINGLE POINT OF FAILURE");
System.out.println();
System.out.println("Fix: Use a managed database with multi-AZ replication");
System.out.println(" Primary in ap-south-1a");
System.out.println(" Synchronous replica in ap-south-1b");
System.out.println(" Automatic failover if primary dies");
}
}
Expected output:
Application tier: 3 pods across 3 zones -> fault tolerant
Database tier: single instance -> SINGLE POINT OF FAILURE
Fix: Use a managed database with multi-AZ replication
Primary in ap-south-1a
Synchronous replica in ap-south-1b
Automatic failover if primary dies
How do you spread replicas across zones?
Topology spread constraints force the scheduler to place replicas in different zones so a whole-zone outage still leaves healthy pods.
package com.bank.deploy;
public class MultiZoneDeployment {
public static void main(String[] args) {
String[] zones = {"ap-south-1a", "ap-south-1b", "ap-south-1c"};
int replicas = 3;
for (int i = 0; i < replicas; i++) {
String zone = zones[i % zones.length];
System.out.println("payment-service-" + (i + 1) + " -> " + zone);
}
System.out.println("\nZone ap-south-1a goes down:");
System.out.println(" payment-service-1: DOWN");
System.out.println(" payment-service-2: HEALTHY (ap-south-1b)");
System.out.println(" payment-service-3: HEALTHY (ap-south-1c)");
System.out.println(" Service continues with reduced capacity.");
}
}
Expected output:
payment-service-1 -> ap-south-1a
payment-service-2 -> ap-south-1b
payment-service-3 -> ap-south-1c
Zone ap-south-1a goes down:
payment-service-1: DOWN
payment-service-2: HEALTHY (ap-south-1b)
payment-service-3: HEALTHY (ap-south-1c)
Service continues with reduced capacity.
What should a senior engineer know?
The complete picture is: run at least three replicas spread across at least three zones, let Kubernetes self-heal back to the desired count, gate traffic with readiness probes, auto-scale to absorb surges, protect maintenance with a PDB, and make the data layer highly available too.
Fault tolerance checklist:
[x] 3+ replicas across 3+ availability zones
[x] Pod anti-affinity to prevent co-location
[x] Liveness + readiness probes configured
[x] HPA for CPU-based auto-scaling
[x] PodDisruptionBudget (minAvailable: 2)
[x] Database with multi-AZ replication + failover
[x] Circuit breaker + bulkhead on downstream calls
Visual Overview
Fault Tolerance Architecture:
Zone ap-south-1a Zone ap-south-1b Zone ap-south-1c
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ payment-svc │ │ payment-svc │ │ payment-svc │
│ pod-1 │ │ pod-2 │ │ pod-3 │
│ node-a │ │ node-b │ │ node-c │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└───────────────────┼───────────────────┘
│
┌────────┴────────┐
│ Load Balancer │
│ (routes to │
│ healthy pods) │
└────────┬────────┘
│
┌────────┴────────┐
│ ReplicaSet │
│ Controller │
│ desired: 3 │
│ actual: 3 │
│ (self-heals │
│ on crash) │
└─────────────────┘
Key Takeaway
If two instances both crash, the real lesson is that two instances were never enough, and the fix is fault tolerance built from redundancy across independent failure domains plus a platform that detects, replaces, and scales automatically. The first check is whether replicas are actually spread out: running two pods on the same node gives almost no protection because one node failure kills both. In Kubernetes, pod anti-affinity and topology spread constraints ensure replicas land on different nodes and different zones.
Kubernetes self-healing is the automatic recovery loop. You declare a desired replica count, say three, and the controller continuously drives the actual count back to that number. When two pods crash, the ReplicaSet creates two replacements with no human involved. Health probes make this precise: a liveness probe decides whether a pod should be killed and restarted, and a readiness probe decides whether it can receive traffic, so freshly created replacements stay out of the load-balancer rotation until warm. Auto-scaling with the HPA adds pods when survivors are overloaded, and a PodDisruptionBudget ensures that voluntary disruptions like node upgrades never take too many pods at once.
At National Bank of India, Pankaj configured the payment service with three replicas across three zones, readiness probes gating traffic until the connection pool warmed, an HPA scaling to ten during month-end surges, a PDB requiring two pods always available, and a multi-AZ database with automatic failover. When two pods crashed during salary processing, the survivors absorbed the load, the controller recreated the lost pods, the HPA added two more, and Rajesh at the counter never noticed.
Remember one sentence: run at least three replicas spread across at least three zones, let Kubernetes self-heal to the desired count, gate traffic with readiness probes, auto-scale with HPA, protect with a PDB, and make the database highly available too.
---