How to estimate target throughput?

• Little's Law – the average number of concurrent requests equals the arrival rate multiplied by the average time each request spends in the system (L = lambda x W).
• Concurrency – for 1000 TPS at 100 ms each, concurrency is 1000 x 0.1 = 100 requests in flight at any instant.
• Transient working set – concurrency times per-request retained memory gives the heap needed for in-flight requests.
• Heap sizing – set -Xmx to two to three times the live set (transient plus steady-state) so the GC has headroom.
• GC selection – match the collector's pause behaviour to the latency budget and its throughput to the allocation rate.

What is Little's Law and why does it matter?

Little's Law says L = lambda x W, where L is the average number of requests in the system, lambda is the throughput (requests per second), and W is the average time in the system (seconds). For 1000 TPS at 100 ms, L = 1000 x 0.1 = 100. On average 100 requests are being processed at any instant. This number drives thread pool size, memory, and GC tuning.

How do you convert concurrency to memory?

If one request retains about 2 MB at peak (request object, domain graph, JSON buffers), then 100 concurrent requests have a transient working set of 200 MB. Add the steady-state heap for caches, connection pools, and reference data, perhaps 300 MB, giving a live set around 500 MB.

package com.bank.capacity;

import java.math.BigDecimal;

public class CapacityPlanner {

    public static void main(String[] args) {
        double tps = 1_000;
        double latencySeconds = 0.100;
        double concurrency = tps * latencySeconds;              // 100

        long perRequestBytes = 2L * 1024 * 1024;                // 2 MB measured
        long transientSet = (long) concurrency * perRequestBytes; // ~200 MB
        long steadyState = 300L * 1024 * 1024;                  // caches, pools
        long liveSet = transientSet + steadyState;               // ~500 MB

        System.out.println("Concurrency: " + (long) concurrency);
        System.out.println("Transient set: " + transientSet / (1024 * 1024) + " MB");
        System.out.println("Live set: " + liveSet / (1024 * 1024) + " MB");
        System.out.println("Recommended -Xmx: " + (liveSet * 3) / (1024 * 1024) + " MB");
    }
}

Concurrency: 100
Transient set: 200 MB
Live set: 500 MB
Recommended -Xmx: 1500 MB

How do you size the heap?

Set maximum heap to two to three times the live set so the collector has room. With a 500 MB live set, choose -Xmx1500m to -Xmx2g. Set -Xms equal to -Xmx on servers to avoid runtime resizing pauses. Too small a heap means frequent collections; too large wastes memory and can lengthen individual pauses.

How do you compute the allocation rate?

Allocation is higher than retained because of intermediate objects. If each request allocates 4 MB, then at 1000 TPS the allocation rate is about 4 GB per second. The GC must reclaim that fast enough.

double allocPerRequestMB = 4;
double allocationRateGBps = (tps * allocPerRequestMB) / 1024;   // ~3.9 GB/s
System.out.println("Allocation rate: " + allocationRateGBps + " GB/s");

How do you choose a garbage collector?

G1GC (default since Java 9) targets a pause-time goal via -XX:MaxGCPauseMillis. For a 100 ms latency budget, set the goal to about 50 ms.

-Xms2g -Xmx2g -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:+ParallelRefProcEnabled

For very low pauses at large heaps, ZGC or Shenandoah keep pauses in the low single-digit milliseconds. The Parallel collector maximizes throughput but has longer stop-the-world pauses, suiting batch jobs rather than latency-sensitive APIs.

| Collector | Pause Target | Best For |
|-----------|-------------|----------|
| G1GC | Configurable (e.g. 50 ms) | General APIs |
| ZGC | Sub-millisecond | Large heaps, strict SLO |
| Shenandoah | Sub-millisecond | Low-latency services |
| Parallel | Maximise throughput | Batch processing |

What happens when latency rises under load?

Latency and throughput are coupled. If latency rises under load, Little's Law says concurrency rises too, which increases memory and can spiral. You protect the system with bounded thread pools and admission control.

What are common mistakes?

The wrong approach is guessing heap and pool sizes without measurement.

# WRONG: random heap, no GC tuning
java -Xmx512m -jar payment-service.jar

The correct approach derives every number from Little's Law and measured per-request footprint.

# CORRECT: derived from capacity plan
java -Xms2g -Xmx2g -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -jar payment-service.jar

How do you validate the estimate?

Run a load test at the target 1000 TPS, watch P99 latency, GC pause time, and allocation rate, and adjust heap and collector flags from real data. Plan headroom for bursts above the average.

What are common interview follow-up questions?

• How does virtual thread scalability change the concurrency calculation?
• What if the service is IO-bound versus CPU-bound?
• How do you size the database connection pool alongside the thread pool?