Definition
- Postmortem -- A written retrospective produced after an incident documenting what happened, the timeline, why it happened, the impact, how it was resolved, and action items.
- Blameless -- Deliberately framed to analyze systemic and process causes rather than assign individual fault.
- Why blameless -- When people fear punishment, they minimize or distort what they did, hiding the very information needed to fix the real cause.
- Human error reframing -- A human mistake is treated as a symptom of a systemic weakness (missing guardrail, confusing tooling), not as the root cause.
- Accountability -- The team is accountable for completing action items; individuals are not blamed for honest mistakes any competent person could have made.
After a serious mix-up at the National Bank of India, the team writes an honest report: what went wrong, when, why, how many customers it hurt, and what changes will stop it happening again. The report never says "Rajesh pressed the wrong button and should be scolded." Instead it asks "why did a system allow a single wrong button press to break transfers, and how do we make that impossible?" Because no one is being blamed, everyone tells the full truth about what they did and saw, and only with that full truth can the bank fix the deeper weakness.
Understanding the Concepts
What is the structure of a blameless postmortem?
A good postmortem has a standard structure so nothing important is lost. The document is shared widely because the learning is valuable beyond the team that experienced the incident.
# Postmortem template:
## Summary
One-paragraph description of what happened.
## Impact
- Duration: 10:28 - 10:55 UTC (27 minutes)
- Users affected: ~25,000 (all transfer-api users)
- Business impact: ~4,500 failed transfers, ~112M INR affected
- Customer complaints: 340 support tickets
## Timeline
- 10:28 - Deploy v5.3 begins rolling out
- 10:30 - Error rate spikes to 40% (detected by Prometheus alert)
- 10:32 - On-call Rajesh acknowledges, opens incident channel
- 10:35 - Triage: SEV1, transfers failing for all users
- 10:38 - IC Pankaj identifies deploy at 10:28 as prime suspect
- 10:40 - Rollback initiated
- 10:42 - Rollback complete, error rate dropping
- 10:55 - Error rate at pre-incident baseline. Incident resolved.
## Root Cause
A database migration in v5.3 dropped the index on the customer_id
column of the accounts table. Without the index, account lookups
changed from index scans (0.05ms) to sequential scans (380ms),
causing timeouts and 5xx errors.
## What Went Well
- Alert fired within 2 minutes of the deploy
- Rollback was fast (2 minutes from decision to completion)
- Communication was clear and timely
## What Got Lucky
- The deploy happened during business hours, not at 3 AM
- The on-call engineer was already at their desk
## Action Items
- [Rajesh, Jan 22] Add index-presence check to CI pipeline
- [Priya, Jan 29] Add canary with auto-rollback for transfer-api
- [Pankaj, Feb 5] Add database query latency alert (>100ms)
- [Rajesh, Feb 12] Add pre-deploy database migration review step
Why is it blameless?
When people fear punishment, they minimize, omit, or distort what they did, which hides the very information needed to understand the failure. A blame culture actively prevents learning.
# With blame:
# Rajesh: "I... I do not know how the index got dropped."
# (He knows, but he is afraid to say he ran the wrong migration file.)
# Root cause: "Unknown. Possibly a configuration error."
# Action items: "Be more careful with migrations."
# Result: vague cause, vague fix, it will happen again.
# Without blame (blameless):
# Rajesh: "I ran the migration script, but I accidentally used the
# wrong file because the naming was confusing -- there were two
# similar files in the same directory."
# Root cause: "Confusing migration file naming + no CI check for
# dropped indexes + no pre-deploy migration review."
# Action items:
# - Rename migration files with clear conventions
# - Add CI check that fails if an index is dropped without approval
# - Add mandatory migration review before production deploys
# Result: three concrete systemic fixes. It cannot happen again.
How does the blameless reframing work?
A blameless approach assumes people acted reasonably given the information and tools they had at the time, and treats a human error as a symptom of a deeper systemic weakness.
# Blame framing (WRONG):
# "Rajesh was careless and dropped the index."
# Fix: "Tell Rajesh to be more careful."
# Problem: you cannot reliably "fix" a person.
# Blameless framing (CORRECT):
# "The system allowed a single migration to drop a production index
# with no guardrail, no confirmation, and no CI check."
# Fix: Add a guardrail, a CI check, and a review step.
# Result: the whole class of failure is prevented.
What organizational benefits does blamelessness produce?
Psychological safety means engineers report incidents and near-misses promptly and describe them fully, which increases the raw material for improvement.
# Benefits:
# 1. Engineers report incidents faster (no fear of punishment)
# 2. Root-cause analysis is deeper (people share honestly)
# 3. Fixes are durable (systemic, not personal)
# 4. Near-misses are surfaced ("I almost dropped the index")
# 5. Culture improves (incidents = learning, not punishment)
# 6. Essential in banking where hiding problems is far more dangerous
# than surfacing them
What about hindsight bias?
A blameless review avoids hindsight bias by judging decisions based on what was known at the time, not on what became obvious only afterward.
# Hindsight bias:
# "Obviously the migration would drop the index. Why did Rajesh not see it?"
# But at the time, Rajesh had 15 migration files with similar names,
# no CI check, and no review process. The mistake was reasonable given
# the tools he had.
# Blameless reframing:
# "Given the confusing naming and lack of safeguards, this mistake was
# predictable. The question is: how do we prevent it structurally?"
What are common mistakes in postmortems?
# WRONG: Good analysis but no completed action items
# The postmortem identifies three fixes.
# Nobody is assigned. No due dates. Six months later, nothing changed.
# The same incident happens again.
# CORRECT: Every action item has an owner and a due date.
# Track completion weekly. Close the postmortem only when all items are done.
# WRONG: Postmortem for every minor issue
# The team writes 20 postmortems per month. Fatigue sets in.
# People stop reading them. Action items pile up.
# CORRECT: Reserve postmortems for incidents above a severity threshold.
# SEV1 and SEV2 get postmortems. SEV3 gets a brief incident review.
What should a senior engineer know?
Blameless does not mean no accountability. The team is absolutely accountable for completing the action items and improving the system. What you remove is personal blame for honest mistakes. A senior engineer also tracks recurring themes across many postmortems to identify systemic patterns worth larger investment.
# Recurring theme analysis:
# Last 10 postmortems:
# - 4 involved missing index or slow query
# - 3 involved no canary rollback
# - 2 involved stale config
# Pattern: invest in CI database checks and canary with auto-rollback.
# One investment prevents multiple classes of incidents.
What are common interview follow-up questions?
- "What if someone was genuinely negligent?" -- Genuine negligence is addressed privately through performance management, not in the postmortem. The postmortem still focuses on systemic fixes.
- "Who facilitates the postmortem?" -- Someone who can keep the discussion on systems and away from finger-pointing, often the IC or a dedicated facilitator.
- "How soon after the incident?" -- Within 1-3 business days, while memory is fresh but emotions have cooled.
Visual Overview
Blameless Postmortem Structure =============================== ┌─────────────────────────────────────────┐ │ 1. SUMMARY What happened (1 para) │ ├─────────────────────────────────────────┤ │ 2. IMPACT Duration, users, cost │ ├─────────────────────────────────────────┤ │ 3. TIMELINE Timestamped events │ ├─────────────────────────────────────────┤ │ 4. ROOT CAUSE Systemic, not personal │ │ "Why did the system allow this?" │ ├─────────────────────────────────────────┤ │ 5. WHAT WENT WELL Good practices │ ├─────────────────────────────────────────┤ │ 6. WHAT GOT LUCKY Near-misses │ ├─────────────────────────────────────────┤ │ 7. ACTION ITEMS │ │ [Owner, Due Date] Concrete fix │ │ [Owner, Due Date] Concrete fix │ │ [Owner, Due Date] Concrete fix │ └─────────────────────────────────────────┘ Key: Focus on SYSTEMS, not PEOPLE. "Why did the system allow this?" not "Who caused this?"
Key Takeaway
A blameless postmortem is a written retrospective produced after an incident that documents what happened, a precise timeline, why it happened, the impact, how it was resolved, and concrete action items to prevent recurrence. It is deliberately framed to analyze systemic and process causes rather than assign individual fault. This is not softness: it is the engineering precondition for getting the honest data that makes the system genuinely safer.
The reason it is blameless is that when people fear punishment, they minimize or distort what they did, hiding the very information needed to find and fix the real cause. A blameless approach treats human error as a symptom of a systemic weakness: if someone ran a command that broke production, the question is not "why were they careless" but "why did the system allow a single command to cause this much damage, and what guardrail, test, or review was missing?" At the National Bank of India, after a transfer outage caused by a dropped index, the postmortem was framed as a systemic gap: the CI pipeline had no index-presence check, and the migration file naming was confusing, so the action items were concrete and owned: a CI check, a canary with auto-rollback, and a database latency alert.
Remember one sentence: a blameless postmortem fixes systems rather than blaming people, because only psychological safety produces the honest account needed to prevent recurrence, and every action item must have an owner and a due date.
---