Definition
- Latency is the time to serve a request, measured separately for successes and failures, as percentiles not averages
- Traffic is the demand on the system: requests per second, bytes per second, or transactions per second
- Errors is the rate of failed requests: explicit (HTTP 500), implicit (wrong content), and policy (too slow)
- Saturation is how full the most constrained resource is relative to its capacity
- Together they tell you whether users are served well and whether the system is heading toward trouble
Think of a teller counter at the National Bank of India serving Rajesh and Priya. Latency is how long each customer waits to be served. Traffic is how many customers arrive per minute. Errors is how many leave without their transaction done. Saturation is how full the counter's capacity is: are all tellers busy, is the queue overflowing, is the cash drawer nearly empty. Watch those four and you know almost everything.
Understanding the Concepts
What is latency and why separate success from failure?
Latency is how long a request takes. The critical refinement is separating successful from failed request latency.
# Success latency p99
histogram_quantile(0.99,
sum by (le) (rate(transfer_duration_bucket{status!~"5.."}[5m]))
)
# Error latency p99 (fast errors can hide problems)
histogram_quantile(0.99,
sum by (le) (rate(transfer_duration_bucket{status=~"5.."}[5m]))
)
# Why separate them?
# A flood of fast 500 errors (2ms each) drags the average DOWN
# Making the latency graph look BETTER during a failure
# Separating ensures errors don't mask the problem
What is traffic?
Traffic measures demand. It is the context for everything else: a spike in errors during a traffic surge tells a different story than the same spike during normal traffic.
# Request rate for the NBI transfer service
sum(rate(transfer_requests_total[5m]))
# A sudden DROP in traffic can itself be a symptom
# (upstream failure stopping requests from reaching you)
What are the three types of errors?
Errors come in three flavours, and capturing all three requires measuring from the client's perspective.
| Error Type | Description | Example |
|---|---|---|
| Explicit | Server returns an error status | HTTP 500 on transfer |
| Implicit | Success status but wrong content | HTTP 200 but empty balance |
| Policy | Technically fine but violates SLO | Transfer took 5 seconds (SLO is 300ms) |
# Explicit errors
sum(rate(transfer_requests_total{status=~"5.."}[5m]))
# Policy errors (latency exceeded SLO threshold)
sum(rate(transfer_requests_total[5m]))
-
sum(rate(transfer_duration_bucket{le="0.3"}[5m]))
What is saturation?
Saturation measures how full the most constrained resource is. It is forward-looking: it predicts near-future problems.
# Database connection pool saturation
hikaricp_connections_active / hikaricp_connections_max
# CPU saturation
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory saturation
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
# Disk IO saturation
rate(node_disk_io_time_seconds_total[5m])
# Saturation is the leading indicator:
# Systems degrade as they approach full capacity
# Latency rises sharply past ~80% saturation
# Connection pool at 100% -> fast-reject errors
How do the four signals work together?
The real value is reading them as a group. Each signal alone tells part of the story; together they tell the whole story.
# Incident at NBI:
# 1. Traffic rises (marketing campaign)
# 2. DB connection pool saturates to 100%
# 3. Latency spikes (waiting for connections)
# 4. Errors burst (pool exhausted, fast rejects)
#
# Reading together: traffic caused saturation,
# saturation caused latency, latency caused errors
# Root cause: connection pool too small for traffic
How are the Golden Signals different from the USE method?
The Golden Signals are request-centric (user-facing). USE (Utilization, Saturation, Errors) is resource-centric. Mature teams use both.
| Framework | Focus | Signals | Use For |
|---|---|---|---|
| Golden Signals | Requests | Latency, Traffic, Errors, Saturation | Service health |
| USE Method | Resources | Utilization, Saturation, Errors | Resource diagnosis |
| RED Method | Requests | Rate, Errors, Duration | Microservices |
What are common mistakes?
The most common mistake is alerting on resource metrics instead of the golden signals.
# WRONG: Alert on cause (CPU high might be fine)
- alert: HighCPU
expr: cpu_usage > 85
# CORRECT: Alert on golden signals (user impact)
- alert: HighTransferLatency
expr: histogram_quantile(0.99, rate(transfer_duration_bucket[5m])) > 0.5
for: 5m
- alert: HighTransferErrorRate
expr: sum(rate(transfer_requests{status=~"5.."}[5m])) / sum(rate(transfer_requests[5m])) > 0.01
for: 5m
What should a senior engineer know?
The Golden Signals map cleanly onto SLIs. Latency and errors are the primary user-facing SLIs. Traffic gives context. Saturation is the leading indicator for capacity. Keep the deeper resource metrics for diagnosis once an alert fires.
What are common interview follow-up questions?
- Why four signals? They cover the user experience (latency, errors), the demand (traffic), and the capacity (saturation) with minimal overlap.
- Why percentiles for latency? Averages hide the slow tail. Use p50, p95, p99.
- What is the most important signal to alert on? Errors and latency, because they directly reflect user experience.
Visual Overview
The Four Golden Signals
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ LATENCY │ │ TRAFFIC │ │ ERRORS │ │SATURATION│
│ How long │ │ How much │ │ How many │ │ How full │
│ │ │ │ │ fail │ │ │
│ p50, p99 │ │ req/sec │ │ 5xx rate │ │ pool % │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
│ │ │ │
└──────────────┴──────┬───────┴──────────────┘
│
Read together:
Traffic ──▶ Saturation
Saturation ──▶ Latency
Latency ──▶ Errors
Key Takeaway
The Golden Signals are the four metrics that together give the most complete picture of a user-facing service: latency, traffic, errors, and saturation. Latency must separate success from failure latency, because a flood of fast errors can make the graph look deceptively healthy. Traffic is the context for everything else. Errors include explicit (HTTP 500), implicit (wrong content), and policy (too slow) failures. Saturation is the leading indicator: the most constrained resource climbing toward full.
The real value is reading them together. In the National Bank of India project, as traffic rose from a marketing campaign, the database connection pool saturated to 100%, which slowed transfers for Rajesh and Priya and caused a burst of fast-reject errors. Because success and failure latency were separate, the fast rejects did not disguise the problem, and saturation pointed straight to the connection pool as the root cause.
The Golden Signals map cleanly onto SLIs: latency and errors are the primary user-facing indicators, traffic gives context, and saturation warns before things break. Alert on these four at the user-facing level and keep the deeper resource metrics for diagnosis once an alert fires.
Remember one sentence: latency and errors are the user-facing signals, traffic gives context, and saturation warns you before things break.
---