Definition
- Root Cause Analysis works backward from symptoms through contributing factors to the underlying systemic causes, refusing to stop at the first trigger
- A blameless post-mortem focuses on how the system allowed the failure, deliberately avoiding naming individuals at fault
- The goal is organizational learning, not punishment: concrete, owned, tracked action items that prevent the same class of failure
- The Five Whys technique moves from person toward system with each iteration
- Action items are categorised as detection (catch faster), prevention (stop it happening), or mitigation (limit damage)
Suppose the National Bank of India accidentally double-charged a batch of transfers one night. A blameful approach asks "who pressed the wrong button" and fires them, which teaches everyone to hide mistakes. The blameless approach asks "how was it possible for one button press to double-charge Rajesh and Priya, and why did nothing catch it?" The fix is not punishing a person; it is fixing the procedure, the test environment, and adding a duplicate-charge safety check.
Understanding the Concepts
What are the phases of incident response?
An incident has distinct phases, and RCA begins once the immediate fire is out.
# Incident response phases:
# 1. DETECT -> Alert fires or user reports
# 2. MITIGATE -> Stop the bleeding (rollback, failover)
# 3. TIMELINE -> Build factual sequence from hard evidence
# 4. RCA -> Work backward through contributing factors
# 5. DOCUMENT -> Write the blameless post-mortem
# 6. TRACK -> Action items with owners and due dates
How do you build the factual timeline?
The timeline is built from hard evidence, not memory, because memory is unreliable under stress.
# NBI Transfer Service Incident Timeline:
# 10:02 Deploy v2.5.1 pushed to production
# 10:05 Prometheus alert: TransferErrorRate > 1% for 2m
# 10:07 On-call engineer Rajesh acknowledged the page
# 10:12 Rajesh identified connection pool config set to 0
# 10:15 Rollback to v2.5.0 initiated
# 10:18 Rollback complete, error rate dropping
# 10:25 Error rate below 0.1%, incident mitigated
# Total impact: 23 minutes, ~12,000 failed transfers
# Sources: deploy logs, Prometheus metrics, PagerDuty,
# Slack messages, git log (not anyone's memory)
How does the Five Whys technique work?
The core of RCA is refusing to stop at the trigger. Each why exposes a deeper layer, moving from the person toward the system.
# Five Whys for the NBI config incident:
#
# Why did transfers fail?
# -> Connection pool was set to 0, no DB connections
#
# Why was the pool set to 0?
# -> A config value was typed wrong in the YAML
#
# Why did a wrong config reach production?
# -> The deploy pipeline has no config validation
#
# Why does the pipeline not validate configs?
# -> We never prioritised adding schema validation
#
# Why was staging not caught it?
# -> Staging uses different config files, not the same ones
#
# Root causes (systemic):
# 1. No config schema validation in the pipeline
# 2. Staging config diverges from production config
# 3. No canary deployment to catch errors before full rollout
What does a blameless post-mortem look like?
Blamelessness assumes every person acted reasonably with the information they had. The real question is what about the system made the mistake easy.
# Post-Mortem: Transfer Service Connection Pool Outage
# Date: 2024-03-15
# Severity: SEV-1
# Duration: 23 minutes
# Impact: 12,000 failed transfers, affecting ~8,000 customers
#
# Summary:
# A configuration change in v2.5.1 set the database connection
# pool to 0, causing all transfer requests to fail.
#
# Root Cause:
# The deploy pipeline has no config validation. A YAML value
# was incorrectly set and reached production unchecked.
#
# Contributing Factors:
# - Staging uses different config files than production
# - No canary deployment to limit blast radius
# - Alert took 3 minutes to fire (scrape interval + for duration)
#
# What Went Well:
# - On-call responded within 2 minutes of page
# - Rollback was fast (3 minutes)
# - Monitoring correctly detected the failure
#
# What Went Poorly:
# - Config reached production without any validation
# - Staging would not have caught this (different configs)
# - 12,000 transfers failed before rollback
What makes good action items?
Action items are the whole point. Each must be specific, assigned, tracked, and categorised.
# Action Items:
#
# PREVENTION (stop it happening):
# [P1] Add JSON schema validation for all config files
# Owner: Pankaj | Due: 2024-03-25 | Ticket: INFRA-4521
#
# [P2] Unify staging and production config templates
# Owner: Priya | Due: 2024-04-01 | Ticket: INFRA-4522
#
# DETECTION (catch it faster):
# [D1] Add synthetic transfer health check (catch in <60s)
# Owner: Rajesh | Due: 2024-03-22 | Ticket: INFRA-4523
#
# MITIGATION (limit damage):
# [M1] Implement canary deployment (5% traffic first)
# Owner: Pankaj | Due: 2024-04-15 | Ticket: INFRA-4524
#
# [M2] Add connection pool minimum validation (reject 0)
# Owner: Rajesh | Due: 2024-03-20 | Ticket: INFRA-4525
What is the difference between blameful and blameless language?
Specific language choices make the difference between a culture of learning and a culture of hiding.
# BLAMEFUL (wrong):
# "Rajesh carelessly deployed a broken config"
# "The developer did not test properly"
# "This would not have happened if someone had checked"
# BLAMELESS (correct):
# "The engineer followed the documented deploy procedure,
# which does not include config validation"
# "The config change passed through the pipeline without
# automated checks because none existed"
# "The test environment uses different config files,
# so this class of error is not caught in staging"
What are common mistakes in post-mortems?
The most common mistake is writing action items that are vague, unowned, or untracked.
# WRONG: Vague action items
# "Improve our deployment process"
# "Be more careful with configs"
# "Add more testing"
# (No owner, no date, no ticket, never done)
# CORRECT: Specific, owned, tracked
# "Add JSON schema validation for deploy configs"
# Owner: Pankaj | Due: March 25 | Ticket: INFRA-4521
What should a senior engineer know?
Run post-mortems for near-misses too, not just customer-facing outages. A near-miss is a free lesson. Share post-mortems widely so other teams learn. Measure whether action items actually get completed. The sign of a mature organization is not having zero incidents but learning permanently from each one.
What are common interview follow-up questions?
- Should every incident get a post-mortem? At minimum, every SEV-1 and SEV-2. Near-misses are free lessons too.
- What if leadership demands blame? Explain that blame causes hiding, which causes worse incidents. Present the evidence.
- How do you prevent post-mortem action items from being ignored? Track them in the backlog like any other work, with owners, due dates, and a regular review.
Visual Overview
Five Whys Depth ┌──────────────────────┐ │ Why 1: What happened │──▶ Transfers failed ├──────────────────────┤ │ Why 2: Proximate │──▶ Pool set to 0 ├──────────────────────┤ │ Why 3: How it passed │──▶ No config validation ├──────────────────────┤ │ Why 4: Why no check │──▶ Never prioritised ├──────────────────────┤ │ Why 5: Why not caught│──▶ Staging diverges └──────────────────────┘ Action Item Categories ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ PREVENTION │ │ DETECTION │ │ MITIGATION │ │ Stop it from │ │ Catch it │ │ Limit the │ │ happening │ │ faster │ │ damage │ │ │ │ │ │ │ │ Config valid.│ │ Health check │ │ Canary deploy│ └──────────────┘ └──────────────┘ └──────────────┘
Key Takeaway
Root Cause Analysis works backward from symptoms through contributing factors to systemic causes, refusing to stop at the trigger. The trigger is the proximate event like a bad config deploy, but the root cause is why that was possible and why nothing caught it. The Five Whys technique moves from person toward system with each iteration, and you expect to find several contributing causes because real incidents happen when multiple missing safeguards line up.
Blamelessness is a specific discipline: every person acted reasonably with the information they had, so the real question is what about the system made the mistake easy. This matters practically because if people fear blame they hide details, and you lose the information needed to fix the system. The post-mortem has a standard shape: summary, quantified impact, timeline, root cause, what went well and poorly, and action items that are specific, assigned to an owner, tracked with a due date.
In the National Bank of India project, a config deploy with a connection pool size of zero took down transfers for twelve thousand customers including Rajesh and Priya for thirty minutes. The Five Whys traced it from failed transfers down to the systemic gap that the pipeline had no config validation and staging used different config files. The team came out with owned, dated action items: config schema validation to prevent, a health check to detect faster, and canary deployment to limit damage.
Remember one sentence: the measure of a mature organization is not having zero incidents but learning permanently from every one, and that only happens when people feel safe enough to tell the whole truth.
---