Definition
- Deliberate failure injection – Intentionally break things in a controlled way to verify the system handles it.
- Steady-state hypothesis – Define what 'normal' looks like so you can detect when the system deviates.
- Controlled experiments – Small blast radius, automatic rollback, monitoring, and a kill switch.
- Proactive discovery – Find weaknesses before they cause real outages, not after.
- Graceful degradation – The expected outcome is that the system bends but does not break.
- Tools – Chaos Monkey, Litmus, Gremlin, AWS Fault Injection Simulator.
Chaos engineering at the National Bank of India is like a fire drill for the bank's systems, except you actually start a small, controlled fire to prove the sprinklers work. Instead of waiting for a server to crash at the worst possible moment, you deliberately kill one on purpose, during business hours with safeguards, and watch whether customers still get served because traffic reroutes to healthy servers. You might slow the network or fill a disk to see if the system degrades gracefully or falls over. The idea, made famous by Netflix's Chaos Monkey randomly switching off servers, is that if you constantly practice surviving failures in a controlled way, real failures become non-events instead of outages.
Understanding the Concepts
What is chaos engineering?
Chaos engineering deliberately injects controlled failures into a system to verify it handles them gracefully.
Traditional testing: "Does it work when everything is fine?"
Chaos engineering: "Does it STILL work when things break?"
What is the steady-state hypothesis?
Before injecting failures, define what "normal" looks like so you can detect deviation.
Steady state for the National Bank of India transfer service:
- Transfer p95 latency < 200ms
- Error rate < 0.1%
- Throughput > 5000 req/s
- All pods healthy
Hypothesis: "If we kill one pod, these metrics should stay
within acceptable bounds because the remaining pods handle
the traffic."
What are the principles of chaos engineering?
- Define steady state : Measurable baseline behavior.
- Hypothesize : Predict what should happen when failure is injected.
- Inject in production : Test where it matters, with real traffic.
- Minimize blast radius : Start small, expand gradually.
- Automate : Run experiments continuously, not just once.
# Litmus chaos experiment definition
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: pod-kill-experiment
spec:
appinfo:
appns: banking
applabel: app=transfer-service
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '60'
- name: CHAOS_INTERVAL
value: '10'
How do you run a chaos experiment safely?
Every experiment has a controlled blast radius, monitoring, and a kill switch.
# Step 1: Verify steady state
curl -s https://nationalbank.in/health | jq .status
# "healthy"
# Step 2: Inject failure (kill one pod)
kubectl delete pod transfer-service-abc123 -n production
# Step 3: Monitor
watch -n 1 'curl -s https://nationalbank.in/metrics | grep p95'
# Step 4: Abort if needed
# Kubernetes auto-restarts the pod
# Load balancer routes to surviving pods
What tools support chaos engineering?
| Tool | Platform | Capabilities |
|---|---|---|
| Chaos Monkey | Netflix/AWS | Random instance termination |
| Litmus | Kubernetes | Pod kill, network, disk, CPU |
| Gremlin | Multi-cloud | Full suite of failure injections |
| AWS FIS | AWS | Managed fault injection service |
| Chaos Mesh | Kubernetes | Network, IO, time, JVM chaos |
Why is chaos engineering important for a bank?
Banks cannot afford unplanned outages. Chaos engineering finds weaknesses proactively.
Without chaos engineering:
"We think the system is resilient."
-> Salary day, one database node fails
-> Transfers stop for 30 minutes
-> 100,000 customers affected
-> Regulatory inquiry
With chaos engineering:
"We tested database node failure last week."
-> Salary day, one database node fails
-> Failover activates in 3 seconds
-> Customers see no impact
-> Confidence maintained
What is the difference between chaos engineering and testing?
Testing verifies expected behavior. Chaos engineering discovers unexpected weaknesses.
Testing: "Does the transfer work with valid inputs?"
Chaos: "Does the transfer survive when the database is slow?"
Testing: known scenarios -> expected results
Chaos: inject failures -> observe actual behavior
(may discover things you did not think to test)
What are common interview follow-up questions?
- "Should you run chaos in production?" Yes, that is where it matters most. But start small with a kill switch and monitoring.
- "What if the experiment causes an outage?" That is a successful experiment: it found a real weakness. Fix it and run again.
- "How do you get management buy-in?" Present the cost of unplanned outages vs the cost of proactive experiments.
Key Takeaway
Chaos engineering is the practice of deliberately injecting controlled failures into a system, often in production, to verify it tolerates real-world faults gracefully rather than failing catastrophically. It rests on the insight that distributed systems fail in complex ways you cannot fully predict, so instead of hoping your resilience mechanisms work, you prove them. The disciplined method is hypothesis-driven: I define a steady state, a measurable indicator of healthy behavior like the transfer success rate and ninety-fifth-percentile latency staying within bounds; I form a hypothesis that the steady state holds even when a specific fault occurs; I inject the fault in a controlled way, killing a server, adding network latency or packet loss, exhausting CPU, memory, or disk, or making a dependency time out; and I observe whether the steady state held. If it held, my confidence in that resilience increases; if it did not, I have found a real weakness to fix before an uncontrolled failure causes an outage. It was popularized by Netflix's Chaos Monkey, which randomly terminated production instances, forcing every service to be built to survive instance loss.
Running it safely is central, so even in production I start with a tiny blast radius like a single instance, run during business hours when engineers are watching, keep strong monitoring to see impact immediately, and have an abort or automatic rollback to stop the moment it causes real harm, often proving the mechanisms in staging first. The value is that chaos tests the parts of resilience normal testing cannot, failover, retries, timeouts, circuit breakers, autoscaling, and graceful degradation, which are notorious for being misconfigured in ways that only show up during a real incident. By injecting the failure deliberately I discover a missing timeout, a retry storm, or a single point of failure in a controlled setting where I can fix it calmly rather than at three in the morning.
In a bank I apply it with extra care because money and compliance are involved, so I prove mechanisms in non-production first, keep a very small blast radius in production with full observability and instant rollback, and avoid any experiment that could corrupt or lose financial data. At the National Bank of India we claimed the transfer service survived losing an instance but had never proven it, so I ran a controlled experiment killing one instance with a success-rate steady state and an abort threshold, which confirmed failover worked on the resilient cluster but, on another service, revealed there was no fallback for a dependency, a weakness we fixed before it hit customers during salary day. Being able to describe chaos engineering as hypothesis-driven controlled fault injection with steady state, blast radius, and abort, and to tie it to Chaos Monkey and to banking-grade safety, is what shows an interviewer I understand it as disciplined resilience verification rather than reckless breakage.
Remember one sentence: chaos engineering proactively injects controlled failures to discover weaknesses before they cause real outages, building confidence that the system degrades gracefully.
---
Visual Overview
Chaos Engineering Process
──────────────────────────
1. Define steady state "Transfer p95 < 200ms, errors < 1%"
│
2. Hypothesize "If we kill one pod, the system
│ should failover without user impact"
│
3. Inject failure Kill the pod (controlled, small blast)
│
4. Monitor Watch metrics, errors, user impact
│
5. Analyze Did the system behave as hypothesized?
│
├── YES -> Confidence increased
└── NO -> Weakness found -> Fix -> Repeat