Definition
- Coordinated omission – a measurement error where the load generator accidentally stops sending during a stall, so slow periods are under-sampled.
- Closed model – a fixed number of virtual users each send-wait-send; a stall delays the next request, so the generator omits requests it should have sent.
- Open model – requests are sent on a fixed schedule regardless of responses; latency is measured from the scheduled send time.
- HdrHistogram correction –
recordValueWithExpectedInterval(value, interval)synthesises the latencies the omitted requests would have had. - Practical impact – p99 reported by a closed model can be a fraction of the true p99 that real users experience.
Imagine measuring how long customers wait at National Bank of India's single teller. In the closed model you have ten testers who each wait patiently during a freeze. In reality, customers keep arriving every few seconds whether or not the teller is frozen, so a huge line forms. The open model keeps customers arriving on schedule during the freeze, capturing the whole backed-up line. Coordinated omission is the closed model politely pausing arrivals during the freeze and then reporting that the bank is fast.
Understanding the Concepts
What is coordinated omission?
In a closed model, you configure N virtual users. Each user runs a loop: send a request, block until the response arrives, think for a moment, then send the next. The next request's send time depends on the previous response's arrival time. If the server stalls for one second, that user simply sends its next request one second later. The tool omits all the requests that a real client population would have sent during that second.
Why does it damage the tail?
Averages and medians survive reasonably, but p99 and p99.9 collapse to fiction. If your service normally answers in two milliseconds but stalls for one second once per thousand requests, a closed model records a handful of slow samples. In reality, if requests arrive at ten thousand per second, a one-second stall means ten thousand requests queued, each experiencing between zero and one second of extra queueing delay.
How does an open model fix it?
In an open model, the load generator sends requests on a fixed schedule regardless of when responses come back. Latency is measured from the scheduled send time, not from when the tool dispatched them. Tools like wrk2, k6 (constant-arrival-rate executor), Gatling (open injection), and Vegeta work this way.
# wrk2: open model at 10,000 requests/second for 60 seconds
wrk2 -t4 -c100 -d60s -R10000 --latency http://localhost:8080/api/balance
How does HdrHistogram correct for coordinated omission?
If a response took one second but you expected one every 100 microseconds, you back-fill the histogram with synthetic latencies for the omitted requests.
import org.HdrHistogram.Histogram;
Histogram histogram = new Histogram(3_600_000_000L, 3);
long latencyUs = 1_000_000; // 1 second measured
long expectedIntervalUs = 100; // expected every 100 us
histogram.recordValueWithExpectedInterval(latencyUs, expectedIntervalUs);
System.out.println("p99: " + histogram.getValueAtPercentile(99.0) + " us");
System.out.println("p99.9: " + histogram.getValueAtPercentile(99.9) + " us");
How do closed and open models compare?
| Aspect | Closed Model (JMeter) | Open Model (wrk2, k6) |
|--------|-----------------------|-----------------------|
| Arrival rate | Depends on response time | Fixed schedule |
| Stall behaviour | Generator pauses | Generator keeps sending |
| Tail latency | Under-reported | Accurately reported |
| Throughput | Self-limiting | Fixed by configuration |
| Use case | Finding saturation point | Validating latency SLOs |
What are common mistakes?
The wrong approach is trusting a closed-model p99 for a latency SLO.
# WRONG: JMeter closed-model p99 = 12 ms
# Production p99 after launch = 800 ms
# Cause: GC pauses hidden by coordinated omission
The correct approach validates with a constant-arrival-rate tool.
# CORRECT: wrk2 at 10,000 RPS shows true p99 = 780 ms
# Now visible: periodic GC stall causing queueing
When should you use a closed model?
A closed model naturally caps throughput at the system's capacity, which is useful for finding the saturation point. Use it for capacity discovery, not for latency SLO validation.
What should a senior engineer know?
The load generator must run on a separate, well-provisioned host. If the generator itself falls behind, you reintroduce coordinated omission on the client side. Monitor the generator's own health during the test.
What are common interview follow-up questions?
- How do you calculate the expected interval for HdrHistogram correction?
- Can you correct for coordinated omission in post-processing?
- How does coordinated omission interact with connection pooling?
Visual Overview
Closed Model (wait-for-response):
User 1: ──REQ──WAIT──REQ──WAIT──(stall 1s)──WAIT──REQ──
User 2: ──REQ──WAIT──REQ──WAIT──(stall 1s)──WAIT──REQ──
▲
Generator pauses here
Omitted requests not measured
Open Model (constant-arrival-rate):
Schedule: ──REQ──REQ──REQ──REQ──REQ──REQ──REQ──REQ──REQ──
▲
Server stalls but requests
keep arriving on schedule
Queueing delay is measured
Impact on Percentiles:
┌───────────────┬──────────┬──────────┐
│ Percentile │ Closed │ Open │
├───────────────┼──────────┼──────────┤
│ p50 │ 2 ms │ 2 ms │
│ p99 │ 12 ms │ 780 ms │
│ p99.9 │ 50 ms │ 990 ms │
└───────────────┴──────────┴──────────┘
Key Takeaway
Coordinated omission is a measurement error where the load generator stops sending requests exactly when the system is slow, so the slow periods are under-sampled and the reported tail latency is far better than reality. It happens in a closed model because the next send time depends on the previous response, so a stall delays arrivals.
The damage lands entirely in the tail. Medians survive but p99 and p99.9 become fiction. An open model fixes this by sending on a fixed schedule regardless of responses, measuring latency from the scheduled send time.
At National Bank of India we signed off a payments endpoint on a JMeter p99 of twelve milliseconds, then saw over eight hundred milliseconds on launch day. The culprit was a periodic GC pause that the closed model hid. Rerunning with wrk2 at a fixed rate exposed the true p99 immediately.
Remember one sentence: production traffic is open, so latency SLOs must be validated with a constant-arrival-rate tool that exposes coordinated omission instead of hiding it.
---