How to design a JVM benchmark?

• Warmup phase – drive the hot methods through C1 and into C2 compilation before recording any measurements.
• GC steady state – run long enough for garbage collection to settle; fix heap size with -Xms equal to -Xmx.
• HdrHistogram – record every latency value into logarithmically sized buckets for accurate high percentiles.
• Isolation – pre-initialise connection pools, caches, and TLS sessions so cold-start costs do not pollute the histogram.
• JMH – Java Microbenchmark Harness: handles warmup, forking, and dead-code prevention via annotations.

Why does the JVM need a warmup phase?

HotSpot runs tiered compilation. A method starts interpreted. As invocation counters cross thresholds, C1 compiles it with profiling counters. When proven hot, C2 recompiles with aggressive optimisations like inlining, escape analysis, and loop unrolling. Measuring before C2 kicks in benchmarks the interpreter, not the production hot path.

# Observe compilation tiers
java -XX:+PrintCompilation -jar benchmark.jar

How do you handle dead-code elimination?

If the benchmark does not use its result, C2 may delete the work entirely. The fix is to consume the result.

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;

@Warmup(iterations = 5)
@Measurement(iterations = 10)
@BenchmarkMode(Mode.SampleTime)
public class InterestBenchmark {

    @Benchmark
    public void calculateInterest(Blackhole bh) {
        BigDecimal interest = principal.multiply(rate);
        bh.consume(interest);  // prevent dead-code elimination
    }
}

How do you account for garbage collection?

Early in a run the heap is small and collections are rare. As caches fill, behaviour changes. Run long enough to see several GC cycles. Fix the heap with -Xms4g -Xmx4g to remove resizing noise. Log GC with -Xlog:gc*.

How do you use HdrHistogram?

Averages hide the tail. HdrHistogram records every value into logarithmically sized buckets.

import org.HdrHistogram.Histogram;

Histogram histogram = new Histogram(3_600_000_000L, 3);

for (int i = 0; i < 100_000; i++) {
    long start = System.nanoTime();
    service.calculateInterest("SAV2001");
    long elapsed = System.nanoTime() - start;
    histogram.recordValue(elapsed / 1000); // microseconds
}

System.out.println("p50: " + histogram.getValueAtPercentile(50.0) + " us");
System.out.println("p99: " + histogram.getValueAtPercentile(99.0) + " us");
System.out.println("p99.9: " + histogram.getValueAtPercentile(99.9) + " us");

How do you isolate the bottleneck?

Pre-initialise everything expensive before the measurement window: open the connection pool and run a warmup query, prime caches, establish TLS sessions.

// Pre-initialise HikariCP before benchmark
HikariDataSource ds = new HikariDataSource(config);
try (Connection c = ds.getConnection()) {
    c.createStatement().execute("SELECT 1");  // warm the pool
}
// Now start measuring

Run the load generator on a separate host. Pin the test to avoid noisy neighbours. Change one variable at a time.

What are common mistakes?

The wrong approach measures a cold JVM.

// WRONG: first iteration includes interpreter + C1 + cold pool
long start = System.nanoTime();
result = service.process(request);
long elapsed = System.nanoTime() - start;
// This number is meaningless for production performance

The correct approach uses JMH with warmup iterations.

// CORRECT: JMH warms up, forks fresh JVM, uses blackhole
@Warmup(iterations = 5, time = 2)
@Measurement(iterations = 10, time = 5)
@Fork(2)
public class SettlementBenchmark { /* ... */ }

What should a senior engineer know?

Warmup must exercise the real code path with realistic data so the JIT profiles the branches production traffic takes. A benchmark answers exactly one question, so decide the question first and design isolation accordingly.

What are common interview follow-up questions?

• How does GraalVM ahead-of-time compilation affect benchmark design?
• How do you benchmark latency versus throughput?
• When is JMH overkill and when is it necessary?