Definition
- Step 1: Confirm impact -- Check user-facing SLIs (error rate, latency, transaction success rate) to size what is broken, for how many users, since when.
- Step 2: What changed -- Check the deploy timeline, config changes, feature flags, and infra changes; the most recent change is the most likely cause.
- Step 3: Mitigate first -- Restore service by the fastest safe means: rollback, flag flip, scale out, failover.
- Step 4: Communicate -- Keep stakeholders and customers informed on a steady cadence.
- Step 5: Root-cause -- Only after service is restored, investigate calmly, write a blameless postmortem.
- Why rollback -- It returns to a known-good state without requiring diagnosis of the new failure.
If a critical alarm sounds at the National Bank of India, you do not first investigate who caused it. You check how bad it really is: are transfers down for everyone or is it a minor glitch? Then you ask what just changed, because a change made minutes ago is usually the culprit and the quickest thing to undo. Then you stop the bleeding by reverting to the version that worked an hour ago, while telling customers what is going on. Only once everything is working again do you sit down to figure out exactly why it broke.
Understanding the Concepts
Step 1: Confirm and size the actual user impact
The first thing to check is the actual user impact, not a single component's status. Open the dashboards and look at user-facing SLIs: error rate, latency, and successful-transaction rate.
# Check user-facing metrics in Grafana/Prometheus
# Error rate:
rate(http_requests_total{app="transfer-api",status=~"5.."}[5m])
/ rate(http_requests_total{app="transfer-api"}[5m])
# Successful transaction rate:
rate(transfer_success_total[5m]) / rate(transfer_total[5m])
# Answer these questions:
# - What exactly is broken? (transfers failing? login down?)
# - For how many users? (100%? 50%? specific region?)
# - Since when? (5 minutes ago? 2 hours?)
# - Getting worse or stabilizing?
# Common trap to avoid:
# "The pod is up, so we are fine."
# But customers are still failing!
# Trust user-facing signals, not component health.
Step 2: Look for what changed most recently
The overwhelming majority of incidents are triggered by a recent change: a deployment, a config change, a feature-flag flip, an infrastructure change, or a sudden traffic shift. A recent change is the most likely cause and the fastest thing to undo.
# Check the deploy timeline
kubectl rollout history deployment/transfer-api -n banking
# Was there a deploy in the last hour?
# Check config and feature-flag changes
kubectl get configmap transfer-api-config -n banking -o yaml
# Were any values changed recently?
# Check ArgoCD sync history
argocd app history transfer-api
# Last sync: 2024-01-15 10:28 (commit abc1234) <- suspect!
# Check infrastructure changes
aws cloudtrail lookup-events --lookup-attributes \
AttributeKey=EventName,AttributeValue=UpdateService \
--start-time 2024-01-15T09:00:00Z
# If the incident started at 10:30 and a deploy happened at 10:28:
# Prime suspect: the deploy.
# You do not need to understand the code yet.
# Proceed to mitigate.
Step 3: Mitigate first to restore service
The fastest safe means to restore service. Rollback is usually best because it returns to a known-good state without diagnosis.
# Option 1: Roll back the deploy
kubectl rollout undo deployment/transfer-api -n banking
# Service restored to the previous version in ~60 seconds.
# Option 2: Flip a feature flag off
curl -X PATCH https://flags.nationalbank.in/api/flags/new-transfer-flow \
-d '{"enabled": false}'
# Feature disabled in ~2 seconds.
# Option 3: Scale out to absorb load
kubectl scale deployment/transfer-api --replicas=10 -n banking
# Option 4: Fail over to a healthy region
# Switch Route53 to point to the secondary region
# Verify mitigation worked:
# Watch error rate drop back to pre-incident baseline
Step 4: Communicate status continuously
Silence during an outage destroys trust faster than the outage itself. Post regular updates even when the update is "still investigating."
# Communication cadence:
# SEV1: Update every 15 minutes
# SEV2: Update every 30 minutes
# Status page update template:
# [2024-01-15 10:35 UTC] INVESTIGATING
# We are aware of issues with the transfer service.
# Customers may experience failures when initiating transfers.
# Our team is actively investigating. Next update in 15 minutes.
# [2024-01-15 10:50 UTC] MITIGATING
# We identified a recent deployment as the likely cause.
# We are rolling back to restore service.
# [2024-01-15 10:55 UTC] RESOLVED
# The rollback has been completed and transfer service is operating normally.
# We will conduct a thorough review and share findings.
Step 5: Root-cause analysis after service is restored
Now that the system is stable, investigate calmly and thoroughly.
# Safe to investigate now:
# 1. Read the diff of the rolled-back deploy
git diff v5.2..v5.3
# 2. Check traces for the failed requests
# In Jaeger: search for error traces between 10:28 and 10:55
# 3. Check database for schema changes
psql -c "SELECT * FROM schema_migrations ORDER BY version DESC LIMIT 5;"
# 4. Write the blameless postmortem with action items
Why is "mitigate before root-cause" the rule?
Two reasons: outages have a direct, continuous cost, and investigating under pressure is slow and error-prone.
# The math:
# Transfer-api handles 10,000 transfers/hour.
# Average transfer: 25,000 INR.
# 1 hour of outage = 10,000 failed transfers = 250M INR affected.
# Rollback takes 60 seconds. Root-cause takes 30-60 minutes.
# Mitigating first saves 29 minutes of outage = ~4,800 transfers.
# The quality:
# Investigating during an outage: stressed, rushing, may make it worse.
# Investigating after rollback: calm, thorough, no customer impact.
Why is rollback usually the fastest safe mitigation?
Rollback returns the system to a state that was known to be working, so it does not require diagnosing the new failure. You simply go back to what worked.
# Rollback does not require understanding the bug:
kubectl rollout undo deployment/transfer-api -n banking
# You do not know what went wrong in v5.3.
# But you know v5.2 was working 30 minutes ago.
# Service restored. Investigate v5.3 safely later.
What are common mistakes?
# WRONG: "Let me understand the bug before rolling back."
# 30 minutes pass. 5,000 transfers fail while you read code.
# CORRECT: Roll back in 60 seconds. Read the code after service is restored.
# WRONG: No communication during the incident
# Customers see failures. No status page update. They call support.
# Support has no information. Trust erodes rapidly.
# CORRECT: Post status updates every 15 minutes, even if just
# "still investigating." Silence is worse than bad news.
What should a senior engineer know?
Mitigation and root cause are genuinely different objectives that must not be conflated. A good responder resists the temptation to "just find the bug first" because a faster restore is almost always achievable than a faster full diagnosis, and the diagnosis loses nothing by waiting until the system is safe. Keeping the previous version warm and deployments reversible makes rollback instantaneous.
Visual Overview
Critical Incident Response Steps
=================================
Incident Reported
│
▼
┌──────────────────────┐
│ 1. CONFIRM IMPACT │ Dashboards, user-facing SLIs
│ What? How many? │ Error rate, latency, tx success
│ Since when? Worse? │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 2. WHAT CHANGED? │ Deploy? Config? Flag? Infra?
│ Most recent change │ Overlay incident time on changes
│ = prime suspect │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 3. MITIGATE FIRST │ Rollback / flag off / failover
│ Fastest safe restore│ Do NOT root-cause yet
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 4. COMMUNICATE │ Status page every 15 min
│ (in parallel) │ Silence destroys trust
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 5. ROOT-CAUSE │ Only AFTER service is stable
│ + POSTMORTEM │ Calm, thorough, blameless
└──────────────────────┘
Key Takeaway
When a critical production incident is reported, the first thing to check is the actual user impact using dashboards and user-facing SLIs, not a single component's status. Size the impact: what is broken, for how many users, since when, and is it getting worse. Then look for what changed most recently, because the overwhelming majority of incidents are triggered by a recent change, and a recent change is both the most likely cause and the fastest thing to undo.
Mitigate first to restore service by the fastest safe means: usually rolling back the last deploy, which returns the system to a known-good state without requiring any diagnosis. Communicate status continuously because silence during an outage destroys trust faster than the outage itself. Only after service is restored do you do deep root-cause analysis and write a blameless postmortem, because investigating a stable system is calm, safe, and thorough while investigating under pressure is slow and error-prone. At the National Bank of India, when customers suddenly could not transfer, the team sized it as SEV1, identified a deploy from two minutes earlier, rolled it back in 60 seconds, and communicated throughout.
Remember one sentence: confirm impact, check what changed, mitigate first by rolling back, communicate status, and root-cause only after service is restored because every minute of outage costs customers.
---