Definition
- Mitigation vs. fix – Separate the fastest safe action to restore service (mitigation) from the durable change that prevents recurrence (longer-term fix), and always mitigate first.
- EC2 + ALB – Remove or replace unhealthy instances via Auto Scaling, then fix the health-check endpoint and tune thresholds and grace periods.
- RDS storage – Grow storage immediately (online, no downtime) or enable storage autoscaling, then add alarms and data-retention policies.
- Failed deployment – Roll back to the last known-good version instantly (blue-green flip, rollout undo), then add staging gates and canary delivery.
- Cardinal rule – Every minute of outage costs customers and money, so restore service before doing deep root-cause analysis.
At the National Bank of India, three different emergencies need two kinds of response each. If some teller counters stop responding (EC2 failing health checks), you first close those counters and open fresh ones (replace instances), then investigate why they froze. If the vault's ledger storage is almost full (RDS out of storage), you first add more shelves immediately (grow storage), then find out why paperwork is piling up. If you rolled out a new process that broke everything (failed deployment), you first revert to yesterday's working process (rollback), then fix the approval checks that let the broken process ship.
Understanding the Concepts
What is the mitigation-first principle?
For any production incident, you separate immediate mitigation from the longer-term fix. Mitigation is the fastest safe action to restore service. The longer-term fix is the durable change that prevents recurrence. You always mitigate first because every minute of outage costs customers and money, and you can investigate safely once the system is stable.
# The pattern for every incident:
# 1. MITIGATE: restore service by the fastest safe means
# 2. COMMUNICATE: tell stakeholders what happened and what you did
# 3. STABILIZE: confirm metrics returned to the pre-incident baseline
# 4. ROOT-CAUSE: investigate why it happened (only after service is restored)
# 5. PREVENT: implement the longer-term fix so it does not happen again
How do you mitigate EC2 instances failing ALB health checks?
The ALB already stops sending traffic to unhealthy instances by marking them as failed. Your job is to ensure the Auto Scaling Group (ASG) replaces them. Set the ASG health check type to ELB so it terminates instances the ALB considers unhealthy and launches fresh ones. If the whole fleet is unhealthy, scale out to add capacity immediately.
# Immediate: Check ALB target health
aws elbv2 describe-target-health \
--target-group-arn arn:aws:elasticloadbalancing:ap-south-1:123456:targetgroup/banking-tg/abc
# If targets are unhealthy, ensure ASG replaces them
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name banking-asg \
--health-check-type ELB \
--health-check-grace-period 300
# Scale out for immediate capacity
aws autoscaling set-desired-capacity \
--auto-scaling-group-name banking-asg \
--desired-capacity 4
# Common root cause:
# Health check hits /health on port 8080 with a 5-second timeout,
# but the Spring Boot app takes 30 seconds to start.
# Every new instance is killed before it finishes booting.
What is the longer-term fix for ALB health-check failures?
Make the health check accurate and resilient. Point it at a lightweight readiness endpoint that returns 200 only when dependencies (database, cache) are ready. Tune the healthy and unhealthy thresholds and the grace period so slow-starting apps are not killed prematurely.
# Longer-term: Fix the health check configuration
aws elbv2 modify-target-group \
--target-group-arn arn:aws:elasticloadbalancing:ap-south-1:123456:targetgroup/banking-tg/abc \
--health-check-path /actuator/health/readiness \
--health-check-interval-seconds 30 \
--health-check-timeout-seconds 10 \
--healthy-threshold-count 2 \
--unhealthy-threshold-count 3
# Also set a grace period long enough for app startup
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name banking-asg \
--health-check-grace-period 120
How do you mitigate RDS running out of storage?
The immediate action is to add storage now. RDS supports online storage increases without downtime for most engines. You can also enable storage autoscaling so it grows automatically, or free space fast by purging old data.
# Immediate: Increase allocated storage (online, no downtime)
aws rds modify-db-instance \
--db-instance-identifier banking-db \
--allocated-storage 200 \
--apply-immediately
# Or enable storage autoscaling
aws rds modify-db-instance \
--db-instance-identifier banking-db \
--max-allocated-storage 500 \
--apply-immediately
# Free space fast: check what is consuming storage
# Connect to the database and check table sizes
psql -h banking-db.abc.ap-south-1.rds.amazonaws.com -U admin -d bankingdb
-- Find the largest tables consuming storage
SELECT schemaname, tablename,
pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC
LIMIT 10;
-- Purge old audit logs that are already archived
DELETE FROM audit_log WHERE created_at < NOW() - INTERVAL '90 days';
VACUUM FULL audit_log;
What is the longer-term fix for RDS storage exhaustion?
Target the growth driver. Enable permanent storage autoscaling with a sensible ceiling, set CloudWatch alarms on FreeStorageSpace well before exhaustion, implement data retention and partition pruning, and review whether a runaway job or unindexed query is writing far more than expected.
# Set a CloudWatch alarm at 20% free storage
aws cloudwatch put-metric-alarm \
--alarm-name banking-db-low-storage \
--metric-name FreeStorageSpace \
--namespace AWS/RDS \
--statistic Average \
--period 300 \
--threshold 20000000000 \
--comparison-operator LessThanThreshold \
--dimensions Name=DBInstanceIdentifier,Value=banking-db \
--evaluation-periods 2 \
--alarm-actions arn:aws:sns:ap-south-1:123456:banking-alerts
How do you mitigate a failed production deployment?
Roll back to the last known-good version immediately. With blue-green, flip traffic back to the stable environment. With Kubernetes, run a rollout undo. Also flip off any new feature flag the release introduced.
# Kubernetes: Rollback to the previous version
kubectl rollout undo deployment/banking-api -n banking
# Verify the rollback
kubectl rollout status deployment/banking-api -n banking
# Verify metrics returned to normal
# Check error rate and latency in Grafana/Prometheus
curl -s http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])
# Blue-green: Flip traffic back to the stable environment
aws elbv2 modify-listener \
--listener-arn arn:aws:elasticloadbalancing:ap-south-1:123456:listener/app/banking-alb/abc/def \
--default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:ap-south-1:123456:targetgroup/banking-blue/abc
What is the longer-term fix for failed deployments?
Close the pipeline gap that let the bad build through. Add automated tests and a staging smoke test that mirrors production, add health gates with automatic rollback on error-rate regression, and use progressive canary delivery so a bad release affects only a small percentage before it is caught.
# ArgoCD Rollout with canary and automatic rollback
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: banking-api
spec:
strategy:
canary:
steps:
- setWeight: 5
- pause: { duration: 5m }
- setWeight: 25
- pause: { duration: 10m }
- setWeight: 50
- pause: { duration: 10m }
analysis:
templates:
- templateName: banking-api-success-rate
args:
- name: service-name
value: banking-api
What are common mistakes in incident mitigation?
# WRONG: Spending 30 minutes diagnosing the root cause while the service is down
# "Let me read the code diff and understand why it broke..."
# Meanwhile, 50,000 customers cannot transfer money.
# CORRECT: Roll back immediately, then investigate
kubectl rollout undo deployment/banking-api -n banking
# Service restored in 60 seconds. Now investigate safely.
# WRONG: Growing RDS storage only after it is 100% full
# At 100%, writes fail, transactions are lost.
# CORRECT: Set an alarm at 20% free so you are warned early
# The alarm fires at 80% used, giving days of runway.
What should a senior engineer know?
A senior engineer knows that mitigation and root cause are different phases with different priorities. They restore service first while communicating status, and begin deep root-cause analysis only once the system is stable. They also know to always keep the previous version warm and deployments reversible, so rollback is always instantaneous. The most important metric during an incident is time-to-restore, not time-to-root-cause.
# Senior practice: always keep the previous version available
# In Kubernetes, the previous ReplicaSet stays scaled down:
kubectl get replicasets -n banking
# NAME DESIRED CURRENT READY
# banking-api-v2-abc 3 3 3 <- current
# banking-api-v1-def 0 0 0 <- previous (rollback target)
# Rollout undo simply scales up v1 and scales down v2
kubectl rollout undo deployment/banking-api -n banking
Visual Overview
Mitigation-First Response Pattern
=================================
Incident Detected
│
▼
┌──────────────────┐
│ 1. MITIGATE │ Restore service FAST
│ (stop bleeding) │ Rollback / replace / grow
└───────┬──────────┘
│
▼
┌──────────────────┐
│ 2. COMMUNICATE │ Status page, Slack, stakeholders
└───────┬──────────┘
│
▼
┌──────────────────┐
│ 3. STABILIZE │ Verify metrics back to baseline
└───────┬──────────┘
│
▼
┌──────────────────┐
│ 4. ROOT-CAUSE │ Investigate AFTER service is stable
└───────┬──────────┘
│
▼
┌──────────────────┐
│ 5. PREVENT │ Longer-term fix to stop recurrence
│ (alarms, gates, │
│ canary, tests) │
└──────────────────┘
Key Takeaway
For any production incident, separate immediate mitigation from the longer-term fix, and always mitigate first because every minute of outage costs customers and money. When EC2 instances fail ALB health checks, let the Auto Scaling Group replace them and scale out for headroom, then fix the health-check endpoint and tune the grace period. When an RDS runs out of storage, grow storage online immediately and free space by purging old data, then make autoscaling and alarms permanent with data-retention policies.
When a production deployment fails, roll back to the last known-good version because rollback is almost always the fastest safe restore. A blue-green flip or a Kubernetes rollout undo restores service in seconds. Then close the pipeline gap with staging smoke tests, canary delivery, and automatic rollback on error-rate regression. At the National Bank of India, when a transfer-service release spiked 5xx errors for transfers involving Rajesh and Priya, the team rolled back immediately and then introduced canary releases with automatic rollback on metric regression.
Remember one sentence: mitigate first to restore service, communicate status, and only do root-cause analysis after the system is stable.
---