Definition
- Kill pod/container – Terminate an instance to verify failover: traffic routes to survivors without user impact.
- Network latency – Add delay between services to test timeout handling, retries, and circuit breakers.
- Disk full – Fill storage to verify alerting, log rotation, and no data corruption.
- Dependency outage – Block an external API to verify fallback: cached responses, degraded mode, or clear error messages.
- Hypothesis + monitor – Every experiment has a stated expected behavior and real-time monitoring.
- Graceful degradation – The system should degrade gracefully in every experiment, never cascade into a full outage.
At the National Bank of India, each chaos experiment pokes a different weak spot to check the system bends instead of breaking. Killing a random server checks another one takes over so customers never notice. Slowing the network checks the system waits sensibly and retries instead of hanging forever. Filling the disk checks the system handles running out of space without corrupting data. Simulating the database going down checks the app shows a polite "try again shortly" and recovers, rather than crashing. Blocking a dependency, say the fraud-check service, checks the app degrades gracefully, maybe queueing or safely declining, instead of taking the whole bank down. The common thread is that a single failure should cause a controlled, limited degradation, never a total outage.
Understanding the Concepts
What are the most common chaos experiments?
Each experiment targets a specific failure mode and has an expected graceful behavior.
| Experiment | Injection | Expected Behavior |
|---|---|---|
| Kill pod | Terminate instance | Failover, no downtime |
| Network latency | Add 500ms delay | Timeouts, retries, circuit breaker |
| Disk full | Fill /var/log | Alert, log rotation, no crash |
| Dependency outage | Block external API | Fallback, cached response |
| CPU stress | Spike to 100% | Throttle, not crash |
| Memory pressure | Allocate large blocks | GC handles, no OOM kill |
How does killing a pod test failover?
Terminating an instance verifies that the system routes traffic to survivors.
# Kill a random pod
kubectl delete pod transfer-service-abc123 -n production
# Expected behavior:
# 1. Kubernetes detects the pod is gone (within seconds)
# 2. Load balancer routes to remaining pods
# 3. Kubernetes starts a replacement pod
# 4. No user-visible impact (requests served by survivors)
# 5. p95 latency stays within SLA
// The application should handle graceful shutdown
@PreDestroy
public void shutdown() {
log.info("Draining in-flight requests before shutdown");
// Complete ongoing transfers, then stop accepting new ones
}
How does adding network latency test resilience?
Injecting delay between services tests timeout handling, retries, and circuit breakers.
# Inject 500ms latency between transfer and accounts service
# Using tc (traffic control) or a chaos tool
tc qdisc add dev eth0 root netem delay 500ms
# Expected behavior:
# 1. Transfer service detects slow responses
# 2. Circuit breaker opens after threshold
# 3. Fallback: "Service temporarily unavailable, try again"
# 4. When latency clears, circuit breaker closes
# 5. Normal service resumes
// Circuit breaker pattern (Resilience4j)
@CircuitBreaker(name = "accountsService", fallbackMethod = "fallback")
public AccountBalance getBalance(String accountId) {
return accountsClient.getBalance(accountId);
}
public AccountBalance fallback(String accountId, Exception e) {
log.warn("Accounts service unavailable, using cached balance");
return cache.getBalance(accountId);
}
How does filling disk space test alerting?
Filling storage verifies the system alerts before it crashes.
# Fill disk to 95%
dd if=/dev/zero of=/var/log/fill bs=1M count=5000
# Expected behavior:
# 1. Monitoring alert fires at 80% threshold
# 2. Log rotation kicks in
# 3. Application continues running
# 4. No data corruption
# 5. Operations team is notified
How does simulating a dependency outage test fallback?
Blocking an external API verifies the system degrades gracefully.
# Block the fraud-check service
iptables -A OUTPUT -d fraud-service.bank.internal -j DROP
# Expected behavior:
# 1. Transfer service cannot reach fraud check
# 2. Circuit breaker activates
# 3. Fallback: queue the transfer for async fraud check
# OR: reject with clear message "Please try again"
# 4. No silent approval of unchecked transfers
# 5. When fraud service recovers, normal flow resumes
What is the expected graceful behavior?
In every experiment, the system should degrade gracefully, never cascade into a full outage.
Graceful degradation:
✓ Affected service is isolated
✓ Other services continue working
✓ Users see a clear, helpful message
✓ Monitoring detects and alerts
✓ Automatic recovery when failure clears
Ungraceful failure (what we are trying to prevent):
✗ One failure cascades to all services
✗ Users see cryptic errors or blank pages
✗ No alerts until customers complain
✗ Manual intervention required to recover
How do you structure an experiment?
Every experiment follows the same template.
Experiment: Kill Transfer Service Pod
──────────────────────────────────────
Hypothesis: "Killing one pod should cause zero customer impact
because Kubernetes routes to surviving pods."
Steady state: p95 < 200ms, errors < 0.1%, all pods healthy
Injection: kubectl delete pod transfer-service-X
Duration: 60 seconds
Blast radius: 1 pod out of 3
Monitoring: Grafana dashboard, PagerDuty alert
Abort: kubectl scale deployment transfer-service --replicas=5
Result: ✓ Hypothesis confirmed / ✗ Weakness found
What are common interview follow-up questions?
- "What is the most important chaos experiment for a bank?" Pod/instance termination, because it tests the most common real-world failure.
- "How often should you run chaos experiments?" Continuously in a Game Day cadence, or automated weekly for critical services.
- "What if the experiment reveals a weakness?" That is a success. Fix the weakness, add a regression test, and re-run the experiment.
Key Takeaway
The common chaos experiments each target a distinct failure mode and the resilience mechanism that should handle it, and in every case the expected outcome is graceful degradation, not total failure. Killing a random pod or container tests self-healing and failover: the orchestrator should detect the loss, the load balancer should stop routing to it, traffic should shift to healthy instances, and a replacement should start, so the success rate barely moves. Adding network latency or packet loss between services tests timeouts and retries: the service should have sensible timeouts so a slow dependency does not hang it, bounded retries with backoff so it does not create a retry storm, and bulkheads so one slow path does not exhaust all threads. Filling disk space tests resource-exhaustion handling: the system should reject new writes cleanly, shed or rotate logs, and alert, rather than corrupting data or crashing. Simulating a database failure tests fallback and recovery: the service should return clear errors or serve safe cache reads, back off and reconnect when the database returns, and never lose committed data. And blocking a dependency tests the circuit breaker and graceful degradation: the circuit should open after repeated failures so calls fail fast with a safe fallback, and the rest of the system should keep working rather than cascading into a total outage.
The unifying expectation is that a single component failure produces a controlled, bounded reduction in capability, failover, fast-failing with fallbacks, clear user-facing errors, and automatic recovery, so customers experience at worst a minor temporary impact. When an experiment instead produces a total failure or a cascade, I have found exactly the weakness to fix, the missing timeout, the unsized pool, the absent circuit breaker, the missing fallback, the naive retry, or the lack of redundancy, before it happens for real. I also always verify recovery, not just failure, because stuck or partial-recovery states are common bugs.
At the National Bank of India I ran exactly this suite. Killing a transfer pod rerouted cleanly, adding latency to the fraud-check call was absorbed by timeouts and a bulkhead, but blocking the fraud dependency on one service revealed there was no circuit breaker, so calls piled up and it cascaded, a total failure we only saw because we injected it deliberately, and we fixed it by adding a circuit breaker and a safe fallback that declined conservatively. I designed every database experiment to simulate unreachability rather than touch data, so financial integrity was never at risk. Being able to map each common experiment, kill a pod, add latency, fill disk, fail the database, block a dependency, to the mechanism it tests and to the expected graceful behavior, and to say that a total failure pinpoints the fix, is what shows an interviewer I understand chaos experiments as targeted resilience verification whose goal is bend, not break.
---
## CHAPTER 21 – QA, TESTING & SDET – COMPLETE! ALL 41 QUESTIONS ANSWERED!
Remember one sentence: each chaos experiment has a hypothesis, a controlled injection, monitoring, and an abort mechanism, and the expected behavior is always graceful degradation, not a cascade.
---
Visual Overview
Common Chaos Experiments ───────────────────────── Experiment │ What You Inject │ Expected Behavior ────────────────────┼───────────────────────┼────────────────── Kill pod/container │ Terminate instance │ Failover, no downtime Network latency │ Add 500ms delay │ Timeouts + retries Disk full │ Fill /var/log │ Alert, log rotation Dependency outage │ Block external API │ Fallback / cached CPU stress │ Spike to 100% │ Throttle, not crash Memory pressure │ Allocate large blocks │ GC handles, no OOM