What incident management is and its five-step process

# The five-step lifecycle:
# 1. DETECT   -> alert fires on user-facing SLIs
# 2. TRIAGE   -> assess severity and scope
# 3. RESPOND  -> assemble team, assign roles, communicate
# 4. FIX      -> mitigate first, root-cause later
# 5. POSTMORTEM -> blameless review with action items

# Prometheus alert rule: transfer API error rate spikes
groups:
  - name: transfer-api-alerts
    rules:
      - alert: TransferAPIHighErrorRate
        expr: |
          rate(http_requests_total{app="transfer-api",status=~"5.."}[5m])
          / rate(http_requests_total{app="transfer-api"}[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Transfer API error rate > 5% for 2 minutes"

# Good detection: alerts on symptoms users feel
# "Error rate > 5%" -> customers are seeing failures

# Bad detection: alerts on internal metrics
# "Pod CPU > 80%" -> may not affect users at all
# Result: alert fatigue, engineers ignore the pager

# Triage questions:
# 1. What exactly is broken? (transfers failing, login down, reports slow)
# 2. For how many users? (all, 50%, specific segment)
# 3. Since when? (5 minutes ago, 2 hours ago)
# 4. Is it getting worse or stabilizing?

# Severity levels at National Bank of India:
# SEV1: Full outage or money movement broken for >25% of users
#       -> page everyone, war room, 15-min status updates
# SEV2: Degraded but working, or <25% of users affected
#       -> page on-call, 30-min status updates
# SEV3: Minor issue, workaround available
#       -> ticket, fix during business hours

# Incident roles:
# Incident Commander (IC): coordinates, makes decisions, does NOT debug
# Responder(s): investigate, diagnose, execute fixes
# Communicator: updates stakeholders, status page, customers

# Example at National Bank of India:
# IC: Pankaj (architect) -> "Rajesh, check the transfer-api logs.
#     Priya, update the status page every 15 minutes."
# Responder: Rajesh -> checks logs, finds the root cause
# Communicator: Priya -> posts updates to status.nationalbank.in

# Automated incident channel creation (PagerDuty/Opsgenie)
# When SEV1 fires:
# 1. PagerDuty pages the on-call (Rajesh)
# 2. Creates Slack channel: #incident-2024-01-15-transfer-api
# 3. Invites IC, responders, communicator
# 4. Posts the alert details and dashboard link

# Mitigation options (fastest first):
# 1. Roll back the last release
kubectl rollout undo deployment/transfer-api -n banking

# 2. Flip a feature flag off
curl -X PATCH https://flags.nationalbank.in/api/flags/new-transfer-flow \
  -d '{"enabled": false}'

# 3. Scale out to absorb load
kubectl scale deployment/transfer-api --replicas=10 -n banking

# 4. Fail over to a healthy region
aws route53 change-resource-record-sets --hosted-zone-id Z123 \
  --change-batch file://failover-to-secondary.json

# Why mitigate before root-cause:
# 1. Every minute of outage costs customers and money
# 2. Investigating under pressure is slow and error-prone
# 3. A stable system can be investigated calmly and thoroughly
# 4. Rollback does not require understanding the bug

# Postmortem structure:
# 1. Summary: one-paragraph description
# 2. Impact: users affected, duration, business impact
# 3. Timeline: timestamped events from detection to resolution
# 4. Root cause: what went wrong and why (systemic, not personal)
# 5. What went well: good practices that helped
# 6. What got lucky: near-misses that could have been worse
# 7. Action items: concrete, owned, with due dates

# Example action items:
# [Rajesh, 2024-01-22] Add index-presence check to CI pipeline
# [Priya, 2024-01-29] Set up canary with auto-rollback for transfer-api
# [Pankaj, 2024-02-05] Add database latency alert to Grafana

# WRONG: No clear roles during the incident
# Everyone debugs at once. Nobody communicates. Nobody decides.
# Two people run conflicting fixes simultaneously.

# CORRECT: Assign IC, responders, communicator immediately.
# IC says: "Rajesh, investigate. Priya, communicate. I coordinate."

# WRONG: Root-causing during the outage
# "Let me read the code diff before rolling back..."
# 30 minutes pass. Customers still cannot transfer.

# CORRECT: Roll back first, investigate later.
# Service restored in 60 seconds. Root cause found in a calm postmortem.

# Incident metrics to track:
# TTD (Time to Detect): how long from failure to alert
# TTA (Time to Acknowledge): how long from alert to human response
# TTM (Time to Mitigate): how long from response to service restored
# TTR (Time to Resolve): how long from detection to full resolution
# Goal: reduce each metric over time through process improvements