Definition
- p99 is the latency below which 99% of requests complete; 1 in 100 is slower than this value
- p95 is the 1-in-20 threshold, p99.9 is the 1-in-1000 threshold, p50 (median) is the typical request
- Averages hide the tail because they are dominated by the many fast requests and say nothing about the distribution
- Fan-out amplification means one slow backend inflates the overall request's p99
- Tightening p99 SLOs is harder because the tail is driven by GC pauses, cold caches, lock contention, and retries
Imagine timing every transfer Rajesh, Priya, and a million other customers make at the National Bank of India. The average transfer time might be 50ms and look wonderful, but the average is hiding the unlucky customers. The p99 asks: "99 out of 100 transfers finish faster than what number?" If the p99 is 2 seconds, one in every hundred customers waits more than 2 seconds, and with a million transfers that is ten thousand frustrated customers. The average is the experience of nobody in particular; the p99 is the real experience of a large, angry group.
Understanding the Concepts
What are latency percentiles?
A percentile is defined by sorting all request latencies and picking the value at a position. p99 means 99% of requests are at or below it.
# Sorted latencies (10 requests):
# 5ms, 8ms, 10ms, 12ms, 15ms, 18ms, 20ms, 25ms, 30ms, 2000ms
#
# p50 (median) = 15ms (typical request)
# p95 = 30ms (1 in 20 is slower)
# p99 = 2000ms (1 in 100 is slower)
# Average = 214ms (misleading! dominated by the outlier)
# Prometheus: compute percentiles from a histogram
# p50 (median)
histogram_quantile(0.50, sum by (le) (rate(transfer_duration_bucket[5m])))
# p95
histogram_quantile(0.95, sum by (le) (rate(transfer_duration_bucket[5m])))
# p99
histogram_quantile(0.99, sum by (le) (rate(transfer_duration_bucket[5m])))
Why do averages hide the tail?
The average is pulled by the many fast requests and says nothing about the distribution. A few fast errors can even drag it down.
# 99 requests at 10ms, 1 request at 2000ms:
# Average = (99 * 10 + 2000) / 100 = 29.9ms (looks great!)
# p99 = 2000ms (reveals the real problem)
#
# The average says "30ms, everything is fine"
# The p99 says "1% of users waited 2 full seconds"
Why does tail latency matter in fan-out systems?
Many real requests fan out to multiple backends and must wait for all of them. The request latency is the slowest backend.
# Transfer request fans out to 5 services:
# Each service has p99 = 100ms (1% chance of being slow)
#
# Probability that ALL 5 are fast:
# 0.99^5 = 95.1%
#
# Probability that AT LEAST ONE is slow:
# 1 - 0.99^5 = 4.9%
#
# The overall request's p95 is now ~100ms
# even though each individual service's p99 is 100ms!
#
# Fan-out amplifies tail latency
// Fan-out in the NBI transfer service
public TransferResult processTransfer(TransferRequest req) {
// Fan out to multiple services in parallel
CompletableFuture<FraudCheck> fraud = fraudService.check(req);
CompletableFuture<Balance> balance = accountService.getBalance(req);
CompletableFuture<Limit> limit = limitService.checkLimit(req);
// Wait for ALL to complete -- latency = slowest one
CompletableFuture.allOf(fraud, balance, limit).join();
// If any one is slow, the whole transfer is slow
}
Why is tightening a p99 SLO harder than tightening the mean?
Improving the mean usually means speeding up the common path, which is often straightforward. The p99 is driven by different, rarer phenomena.
| Tail Cause | Why It Is Hard to Fix |
|---|---|
| GC pauses (stop-the-world) | Requires GC tuning or different collector |
| Cold caches | First request after eviction is always slow |
| Lock contention | Requires architectural change |
| Queueing at saturation | Requires capacity or backpressure |
| Retries | Amplified by timeouts and fan-out |
| Slowest fan-out dependency | Requires hedged requests or isolation |
# Improving the mean: add a cache
# Before: mean = 50ms, p99 = 2000ms
# After: mean = 10ms, p99 = 1800ms (barely moved!)
# The cache helps the common case but not the tail
#
# Improving the p99: hedge the slow dependency
# Send the request to two replicas, take the first response
# Before: p99 = 2000ms
# After: p99 = 200ms (tail dramatically reduced)
How do you measure percentiles correctly?
Never average percentiles across servers or time buckets. Percentiles are not additive. Compute them from the underlying histogram.
# WRONG: Average the p99 from each server
# Server A p99 = 100ms, Server B p99 = 500ms
# "Average p99" = 300ms <- MEANINGLESS
# CORRECT: Merge the histograms, then compute p99
# Prometheus does this with sum by (le)(rate(bucket[5m]))
# then histogram_quantile(0.99, ...)
What are common mistakes?
The most common mistake is reporting only the average and missing tail problems entirely.
# WRONG dashboard:
# "Average latency: 30ms" (hides the tail)
# CORRECT dashboard:
# p50: 15ms | p95: 80ms | p99: 500ms | p99.9: 2000ms
# Shows the full distribution at a glance
What should a senior engineer know?
You choose which percentile to put in your SLO based on how sensitive users are to the tail. For a payment confirmation you care about p99 and even p99.9. For a background report the mean may suffice. Each step tighter exposes rarer causes and costs more to fix.
What are common interview follow-up questions?
- What is the difference between p99 and max? p99 ignores the absolute worst outlier; max is the single slowest request ever. Max is noisy and unreliable for SLOs.
- What causes high p99 but low average? GC pauses, cold caches, lock contention, slow fan-out dependencies.
- Can you aggregate percentiles? No, you must aggregate histograms first, then compute the percentile.
Visual Overview
Percentile Distribution
Latency (ms)
2000 | * (p99)
| *
500 | *
| *
100 | * * * (p95)
50 | * * * * * *
10 | * * * * * * (p50)
└────────────────────────────────────────────
0% 50% 90% 95% 99% 100%
Percentile rank
Fan-out Amplification
┌──────────┐
│ Transfer │──▶ Fraud (p99=100ms)
│ Request │──▶ Account (p99=100ms)
│ │──▶ Limit (p99=100ms)
│ │──▶ Notify (p99=100ms)
│ │──▶ AML (p99=100ms)
└──────────┘
Wait for ALL: p95 of overall = ~100ms
(1 - 0.99^5 = 4.9% chance of at least one being slow)
Key Takeaway
A latency percentile is the value below which a given fraction of requests complete. The p99 is the latency that 99% of requests finish within, meaning 1 in 100 is slower. The p50 is the typical request and each higher percentile looks further into the slow tail. You track these instead of the average because the average is dominated by the many fast requests and tells nothing about the distribution: 99 requests at 10ms and 1 at 2 seconds produces a healthy-looking 30ms average that masks the problem.
Tail latency matters most at scale and in fan-out systems. Many requests fan out to multiple backends and must wait for all of them, so the overall latency is the slowest backend. At the scale of millions of requests, the p99 is not a curiosity: it is tens of thousands of users per window.
Tightening an SLO on p99 is genuinely harder than on the mean. Improving the mean usually means speeding up the common path with a cache or better query. But the p99 is caused by rare, correlated events: GC pauses, cold caches, lock contention, retries, and the slowest fan-out dependency. In the National Bank of India project, the transfer service had a beautiful average latency, but 1 in 100 transfers for Rajesh and Priya took nearly 2 seconds, driven by fan-out where any one slow backend dragged the whole transfer.
Remember one sentence: the average is the experience of nobody in particular, while the p99 is the real experience of a large group of users that the average conveniently buries.
---