Definition
- Error budget is the maximum allowed unreliability over a window, derived as 100% minus the SLO target
- If the SLO is 99.9%, the budget is 0.1% of requests that may fail before the SLO is breached
- While budget remains, the team ships features at pace and takes reasonable risks
- When budget is exhausted, non-essential releases freeze and the team fixes reliability
- Burn-rate alerts page immediately on a fast burn and raise a gentle ticket on a slow burn
The National Bank of India promises Rajesh and Priya that 99.9% of transfers will succeed. That leaves a tiny allowance of 0.1% that are permitted to fail: the error budget, like a monthly spending limit on unreliability. When the team ships a new feature and it causes a few failures, they are spending the budget. As long as budget remains, they keep shipping. But if a bad release burns the whole allowance in a week, the bank says "stop launching, fix stability first."
Understanding the Concepts
How is the error budget derived?
The derivation is simple arithmetic. If the SLO says 99.9% of requests must succeed over 28 days, then 0.1% may fail. That 0.1% is the error budget.
# Error budget derivation:
# SLO: 99.9% success over 28 days
# Budget: 100% - 99.9% = 0.1%
# Total requests in window: 10,000,000
# Allowed failures: 10,000,000 x 0.001 = 10,000
# As downtime for a pure availability SLO:
# 99.9% monthly = ~43 minutes allowed downtime
# 99.99% monthly = ~4.3 minutes allowed downtime
How does the budget govern releases vs reliability?
The budget turns the endless argument between developers who want to ship and SREs who want stability into a data-driven policy.
# Error Budget Policy (agreed by eng + product):
#
# Budget remaining > 50%:
# -> Ship features at normal pace
# -> Take reasonable risks (migrations, experiments)
#
# Budget remaining 20-50%:
# -> Ship features cautiously
# -> All deploys require canary + rollback plan
#
# Budget remaining < 20%:
# -> Freeze non-essential releases
# -> Team redirects to reliability work
#
# Budget exhausted (0%):
# -> Full release freeze
# -> Only critical fixes and reliability improvements
# -> Releases resume when rolling window restores budget
How does planned vs unplanned spending work?
Budget is spent through both planned risks and unplanned incidents.
# NBI transfer service error budget: 10,000 failures / 28 days
#
# Day 1-7: Feature release burns 3,000 (planned risk)
# Day 8: Dependency outage burns 5,000 (unplanned)
# Day 9-10: DB failover burns 2,500 (unplanned)
# Total: 10,500 -- budget EXHAUSTED
#
# Release gate automatically freezes non-essential deploys
# Team stops shipping features to fix regressions
# Budget recovers as the 28-day window rolls forward
How do burn-rate alerts work?
Burn rate measures how fast the budget is being consumed. Multi-window multi-burn-rate alerts page based on severity.
# Fast burn: would exhaust monthly budget in 1 day
- alert: TransferErrorBudgetFastBurn
expr: |
(
sum(rate(http_requests{status=~"5.."}[1h]))
/ sum(rate(http_requests[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests{status=~"5.."}[5m]))
/ sum(rate(http_requests[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Fast error budget burn -- would exhaust in <1 day"
# Slow burn: would exhaust monthly budget in ~10 days
- alert: TransferErrorBudgetSlowBurn
expr: |
(
sum(rate(http_requests{status=~"5.."}[6h]))
/ sum(rate(http_requests[6h]))
) > (3 * 0.001)
for: 1h
labels:
severity: warning
annotations:
summary: "Slow error budget burn -- create a ticket"
How is error budget different from error rate?
| Aspect | Error Rate | Error Budget |
|---|---|---|
| What | Current failure percentage | Allowed failures over a window |
| Time frame | Instantaneous / short window | Rolling 28-day window |
| Drives | Immediate alerting | Feature vs reliability decisions |
| Example | 0.5% errors right now | 3,000 of 10,000 failures remaining |
What are common mistakes?
The most common mistake is an error budget policy with no teeth: leadership agrees to it but ignores it when the budget runs out.
# WRONG: Budget exhausted but team keeps shipping
# "We know the budget is gone but this feature is important"
# -> Customers suffer, SLA breach becomes likely
# CORRECT: Policy has real teeth, agreed in advance
# Budget exhausted -> release freeze enforced automatically
# Only critical fixes pass the release gate
What should a senior engineer know?
A team that never touches its error budget has set its SLO too high and is over-investing in reliability at the expense of feature velocity. The budget is intentionally meant to be spent.
# SLO too high: budget never spent
# SLO 99.999% -> budget = 100 failures/month
# Team ships nothing because any release risks breaching
# -> Lower the SLO to match what users actually need
# SLO just right: budget spent but not exhausted
# SLO 99.9% -> budget = 10,000 failures/month
# Team ships features, spends ~60% of budget, has room
# -> Healthy balance of velocity and reliability
What are common interview follow-up questions?
- What happens when the budget recovers? Releases resume as the rolling window moves forward and old failures drop off.
- Is 100% SLO possible? Technically yes but it means zero error budget, so no releases are ever safe.
- Who owns the error budget? Engineering and product jointly, with an agreed policy.
Visual Overview
Error Budget Lifecycle
┌──────────────┐
│ SLO: 99.9% │
│ Budget: 0.1% │──── 10,000 allowed failures / 28 days
└──────┬───────┘
│
▼
┌──────────────┐ Feature ┌──────────────┐
│ Budget > 50% │ releases │ Ship at pace │
│ │────────────▶│ │
└──────┬───────┘ └──────────────┘
│
▼
┌──────────────┐ Cautious ┌──────────────┐
│ Budget 20-50%│ deploys │ Canary + plan│
│ │────────────▶│ │
└──────┬───────┘ └──────────────┘
│
▼
┌──────────────┐ Freeze ┌──────────────┐
│ Budget < 20% │────────────▶│ Fix reliab. │
│ or exhausted │ │ only │
└──────────────┘ └──────────────┘
Key Takeaway
An error budget is the maximum allowed unreliability over a window, derived directly from the SLO as 100% minus the target. If the SLO is 99.9% over 28 days and the service handles ten million requests, the budget is ten thousand allowed failures. The crucial idea is that choosing an SLO below 100% deliberately creates a resource you are allowed to spend.
The budget governs the pace of releases versus reliability work with a single data-driven rule. While budget remains, the team ships features and takes reasonable risks. When the budget is exhausted, releases freeze and the team fixes reliability until the rolling window restores it. Burn-rate alerts page immediately on a fast burn and raise a gentle ticket on a slow burn.
In the National Bank of India project, the transfer service had a 99.9% SLO giving an error budget of ten thousand allowed failures. A feature release burned three thousand, a dependency outage burned five thousand more triggering a fast-burn page, and a database failover pushed past the allowance. The release gate automatically froze non-essential deploys and the team stopped shipping features to fix the regressions.
Remember one sentence: the error budget is meant to be spent, and the moment it runs out, reliability becomes the only priority until it comes back.
---