Definition
- Response time – End-to-end duration of a single request. The most direct measure of user-perceived speed.
- Throughput – Requests per second the system completes. Measures capacity.
- Error rate – Percentage of requests that fail by error status, timeout, or dropped connection. Gates latency validity.
- Latency percentiles – p50 (median, typical), p95 (fairly bad), p99 (slow tail). Describe the distribution shape.
- Why not averages – Averages hide outliers. If 99 users wait 50ms and 1 waits 10s, the average looks fine but p99 reveals the real problem.
- SLA format – "p95 under 200ms with error rate under 1% at 500 req/s."
When the National Bank of India measures its transfer service under load, response time is how long one customer waits, throughput is how many transfers per second, and error rate is the share that fail. The average is misleading: if ninety-nine customers wait fifty milliseconds and one waits ten seconds, the average still looks fine, but that one furious customer is the ninety-ninth percentile, the real experience you must control.
Understanding the Concepts
What is response time?
Response time is the end-to-end duration of a single request from send to full response. You rarely report a single number because requests vary.
// k6: response time is measured automatically
export default function () {
const res = http.post(`${__ENV.BASE_URL}/api/transfer`,
JSON.stringify({ from: 'ACC-001', to: 'ACC-002', amount: 1000 }),
{ headers: { 'Content-Type': 'application/json' } }
);
// k6 records: http_req_duration for every request
// Reports: min, max, avg, med, p90, p95
}
What is throughput?
Throughput is the rate of completed requests, typically requests or transactions per second. It measures capacity.
Throughput and response time interact:
Throughput │ ┌──────── Saturation point
(req/s) │ /│
│ / │
│ / │
│ / │ After saturation:
│/ │ throughput plateaus or drops
└─────┴──────────────────
Load (concurrent users)
Below saturation: throughput grows with load
At saturation: throughput plateaus
Above saturation: throughput drops, latency spikes
What is error rate?
Error rate is the fraction of requests that fail. It is critical because a system that is fast only because it rejects requests is not performing.
// k6: thresholds combine latency AND error rate
export const options = {
thresholds: {
http_req_duration: ['p95<200'], // Latency SLA
http_req_failed: ['rate<0.01'], // Error rate gate
// Both must pass: fast AND correct
},
};
What are latency percentiles?
Percentiles describe the shape of the response-time distribution.
- p50 (median) : The typical experience. Half of requests are faster, half slower.
- p95 : 95% of requests are faster. Captures a fairly bad experience.
- p99 : 99% of requests are faster. Captures the slow tail.
Response time distribution:
Count │ ┌─┐
│ ┌┤ ├─┐
│ ┌┤ │ ├─┐
│ ┌┤ │ │ ├─┐
│ ┌┤ │ │ │ ├─┐ ........ ┌─┐
│ │ │ │ │ │ │ │ │ │ <- slow tail
└─┴─┴─┴─┴─┴─┴─┴─ ─ ─ ─ ─ ┴─┴──
50ms 200ms 10s
p50 p95 p99
Average: ~80ms (looks fine!)
p99: 10s (one in 100 users waits 10 seconds)
The average HIDES the p99 problem.
Why are percentiles better than averages?
Averages are dominated by the bulk and hide outliers. In a high-traffic bank, one percent of requests is a large absolute number.
// Java: demonstrating why averages lie
package com.bank.perf;
import java.util.*;
import java.util.stream.IntStream;
public class PercentileDemo {
public static void main(String[] args) {
// 99 requests at 50ms, 1 request at 10000ms
List<Long> times = new ArrayList<>();
IntStream.range(0, 99).forEach(i -> times.add(50L));
times.add(10000L);
double average = times.stream()
.mapToLong(Long::longValue).average().orElse(0);
Collections.sort(times);
long p50 = times.get(49); // Median
long p95 = times.get(94); // 95th percentile
long p99 = times.get(98); // 99th percentile
System.out.println("Average: " + average + "ms");
System.out.println("p50: " + p50 + "ms");
System.out.println("p95: " + p95 + "ms");
System.out.println("p99: " + p99 + "ms");
}
}
Average: 149ms (looks acceptable!)
p50: 50ms (typical user is fine)
p95: 50ms (most users are fine)
p99: 10000ms (1 in 100 waits TEN SECONDS)
At 10,000 req/s, p99 = 100 users/second waiting 10 seconds.
That is 6,000 frustrated users per minute.
How do you read metrics together?
A service-level objective typically states a percentile latency bound at a given throughput with a maximum error rate.
SLA: "p95 < 200ms, error rate < 1%, at 5000 req/s" Load Level │ p95 (ms) │ p99 (ms) │ Throughput │ Error Rate │ SLA Met? ───────────┼──────────┼──────────┼────────────┼────────────┼───────── 2000 users │ 45 │ 80 │ 4500 req/s │ 0.0% │ Yes 5000 users │ 120 │ 250 │ 5200 req/s │ 0.3% │ Yes 8000 users │ 380 │ 1200 │ 5100 req/s │ 2.5% │ NO 10000 users│ 950 │ 5000 │ 4800 req/s │ 8.0% │ NO Breaking point: ~6000-7000 users Action: diagnose server metrics at that load level
What is the trap of averaging percentiles?
Averaging percentiles across time windows or servers is mathematically invalid. You must compute percentiles from the full distribution.
Wrong: p95 of server A = 100ms, p95 of server B = 200ms
Average p95 = 150ms (INVALID)
Correct: Merge all response times from both servers
Sort the combined distribution
Compute p95 from the merged set
What are common interview follow-up questions?
- "Which percentile matters most?" p99, because it captures the slow tail that real users experience and that averages hide.
- "When is the average acceptable?" Almost never for latency SLAs. Use it only for throughput (average req/s is fine).
- "How do you handle bimodal distributions?" Break them down by endpoint or transaction type, because combining different operations masks each one's real performance.
Visual Overview
Four Key Performance Metrics ──────────────────────────── ┌──────────────────────────────────────────┐ │ 1. RESPONSE TIME │ │ How long one request takes │ │ Report as percentiles, not average │ ├──────────────────────────────────────────┤ │ 2. THROUGHPUT │ │ Requests per second completed │ │ Measures capacity │ ├──────────────────────────────────────────┤ │ 3. ERROR RATE │ │ % of failed requests │ │ Gates validity of latency numbers │ ├──────────────────────────────────────────┤ │ 4. LATENCY PERCENTILES │ │ p50 = typical (50th percentile) │ │ p95 = fairly bad (95th) │ │ p99 = slow tail (99th) <- MOST IMPORTANT│ └──────────────────────────────────────────┘ Read together: "p95 < 200ms, error < 1%, at 5000 req/s"
Key Takeaway
The four key performance metrics are response time, throughput, error rate, and latency percentiles. Response time is how long one request takes and is the most direct measure of user-perceived speed. Throughput is requests per second and measures capacity. Error rate gates the validity of latency: fast-but-failing is not success. And latency percentiles, p50, p95, and p99, describe the distribution shape, with p99 capturing the slow tail that averages hide.
Percentiles matter more than averages because averages are dominated by the bulk. If ninety-nine users wait fifty milliseconds and one waits ten seconds, the average is 149ms, which looks fine, but the p99 reveals that one in a hundred users waits ten seconds. At ten thousand requests per second, that is a hundred frustrated users every second.
I read metrics together against defined SLA targets: "p95 under 200ms with error rate under 1% at 5000 requests per second." I correlate client-side metrics with server-side resource metrics so a latency spike can be traced to its cause.
At the National Bank of India, the average transfer latency looked healthy during salary day, but the p99 was several seconds because of a saturating connection pool, which only the percentile view revealed.
Remember one sentence: report percentiles not averages because the average hides the slow tail that real users feel, and the p99 is more important than the mean.
---