What an error budget is and how teams use it

# Error budget calculation:
# SLO = 99.9%
# Error budget = 100% - 99.9% = 0.1%

# In requests (1,000,000 requests/month):
# Allowed failures = 1,000,000 * 0.001 = 1,000 failures

# In downtime (30-day month):
# Allowed downtime = 30 days * 24 hours * 0.001 = 0.72 hours = ~43 minutes

# Track budget consumption in Prometheus
# Budget consumed = observed failures / allowed failures
# PromQL:
(
  sum(increase(http_requests_total{app="transfer-api",status=~"5.."}[30d]))
  /
  (sum(increase(http_requests_total{app="transfer-api"}[30d])) * 0.001)
) * 100
# Result: percentage of error budget consumed (e.g., 45%)

# Error budget policy at National Bank of India:
#
# Budget consumed < 50%:
#   -> Ship features freely, take reasonable deployment risk
#   -> Run experiments, A/B tests
#
# Budget consumed 50-80%:
#   -> Ship only low-risk changes
#   -> Tighten canary analysis, extend pause durations
#   -> Review recent incidents for patterns
#
# Budget consumed > 100%:
#   -> FEATURE FREEZE: only reliability fixes and critical security patches
#   -> Mandatory postmortem for each incident that consumed budget
#   -> Focus: fix root causes, improve tests, harden pipeline
#   -> Resume features only when rolling window restores budget

# Multi-window burn rate alerts
# Fast burn: consuming 14.4x the sustainable rate over 1 hour -> PAGE
- alert: TransferApiFastBurn
  expr: |
    (
      1 - sum(rate(http_requests_total{app="transfer-api",status=~"2.."}[1h]))
          / sum(rate(http_requests_total{app="transfer-api"}[1h]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning 14x faster than allowed (1h window)"

# Slow burn: consuming 3x the sustainable rate over 3 days -> TICKET
- alert: TransferApiSlowBurn
  expr: |
    (
      1 - sum(rate(http_requests_total{app="transfer-api",status=~"2.."}[3d]))
          / sum(rate(http_requests_total{app="transfer-api"}[3d]))
    ) > (3 * 0.001)
  for: 1h
  labels:
    severity: warning

# Burn rate alerting tiers:
# 14.4x burn over 1h  -> page on-call immediately (active incident)
# 6x burn over 6h     -> page within 30 minutes
# 3x burn over 3 days -> ticket, investigate during business hours
# 1x burn over 30 days -> everything is fine, budget is sustainable

# Without error budget:
# Product: "Ship the new loan feature NOW!"
# Ops: "No, the service is fragile, we need stability!"
# Result: whoever has more authority wins. No data.

# With error budget:
# Budget remaining: 60%
# Product: "Ship the loan feature."
# Ops: "Budget is healthy. Go ahead."
# Both sides agree because the data says it is safe.

# Budget exhausted: 105%
# Product: "Ship the loan feature."
# Ops: "Budget is spent. Policy says feature freeze until restored."
# Product: "Understood. Let us fix reliability first."
# The decision is made by data, not authority.

# Deployment risk budget model:
# Each deployment has an expected failure rate (e.g., 2% of canary requests fail)
# With 60% budget remaining (600 allowed failures left):
#   A canary affecting 5% of 1M requests = 50,000 requests
#   Expected failures: 50,000 * 2% = 1,000 failures
#   Budget after deploy: 600 - 1,000 = OVER BUDGET
#   Decision: this deploy is too risky. Fix the failure rate first.

# WRONG: Treating the error budget as a stick to punish people
# "Your team exhausted the budget, so you are in trouble."
# Result: teams hide incidents and under-report failures.

# CORRECT: Treat the budget as a steering tool
# Exhausting the budget is a signal, not a punishment.
# Consistently exhausting it means: invest in reliability or relax the SLO.

# WRONG: Setting the SLO so tight that the budget is always exhausted
# SLO = 99.99% but infrastructure only supports 99.95%
# Budget is permanently spent. Team is always in feature freeze.
# Morale drops. Nobody ships anything.

# CORRECT: Set the SLO to match what users need and infrastructure supports.
# If 99.9% is sufficient for users, do not set 99.99%.
# A realistic SLO gives a usable budget.