How to size a thread pool?

• CPU-bound – pure computation (interest calculation, risk scoring). The thread almost never gives up the core. Optimal size: cores + 1. The plus one covers the rare moment a thread is paused by a page fault.
• IO-bound – waiting on disk, network, or database. The thread spends most of its life blocked. Optimal size: cores * (1 + waitTime / computeTime). If a task waits 90ms and computes 10ms, the ratio is 9, so on 8 cores: 8 * (1 + 9) = 80 threads.
• Oversizing – too many threads cause context switching (CPU cycles wasted saving/restoring registers), memory waste (each thread reserves ~1 MB stack), and cache pollution (each thread evicts useful data from CPU cache).
• Bounded queue – always pair the pool with a bounded queue and rejection policy. An unbounded queue hides pool saturation.
• Measure – formulas are starting points. The only reliable approach is to set a size, run a realistic load test, watch CPU utilization and throughput, and adjust.

How do you size a thread pool?

The core idea is that a CPU core can only run one thread at a time. If your task never blocks, adding more threads than cores gives no more parallelism and only wastes time on context switching.

What is the formula for CPU-bound work?

For tasks that do pure computation (interest calculation, risk scoring, encryption), the optimal pool size is the number of CPU cores plus one.

package com.bank.concurrency;

public class CPUBoundSizing {
    public static void main(String[] args) {
        int cores = Runtime.getRuntime().availableProcessors();
        int poolSize = cores + 1;  // e.g., 8 cores → 9 threads
        System.out.println("CPU-bound pool size: " + poolSize);
    }
}

The plus one exists so that if one thread is briefly paused by a page fault or GC safepoint, another is ready to keep the core busy.

What is the formula for IO-bound work?

For tasks that spend most of their time waiting on external IO (database queries, HTTP calls, file reads), the formula accounts for the wait-to-compute ratio.

package com.bank.concurrency;

import java.util.concurrent.*;

public class IOBoundSizing {
    public static void main(String[] args) {
        int cores = Runtime.getRuntime().availableProcessors();
        long waitTime = 90;    // ms waiting for DB
        long computeTime = 10; // ms actual computation
        int poolSize = cores * (int)(1 + waitTime / computeTime);
        // 8 cores * (1 + 9) = 80 threads

        ThreadPoolExecutor pool = new ThreadPoolExecutor(
                poolSize, poolSize, 60L, TimeUnit.SECONDS,
                new ArrayBlockingQueue<>(500),
                new ThreadPoolExecutor.CallerRunsPolicy());
        System.out.println("IO-bound pool size: " + poolSize);
    }
}

Why does oversizing hurt?

Oversizing causes three concrete problems.

Context switching: with 500 threads on 8 cores, the OS constantly swaps threads, wasting CPU cycles on saving and restoring registers. Each context switch costs 5-10 microseconds, and at high frequency this becomes a significant percentage of total CPU time.

Memory: each platform thread reserves approximately 1 MB of stack space. 500 threads reserve 500 MB of virtual memory, reducing heap space available for application data.

Cache pollution: each thread has its own working set of data in CPU cache. Constant switching evicts useful data from L1/L2 cache, forcing expensive main memory accesses when the thread resumes.

What is Little's Law for pool sizing?

Little's Law provides an alternative perspective: concurrency = throughput x latency. If you need 100 requests per second and each takes 500ms, you need at least 100 x 0.5 = 50 concurrent threads.

National Bank of India statement generation:
  Target throughput: 200 statements/second
  Average latency: 400ms per statement
  Required concurrency: 200 x 0.4 = 80 threads

How do you measure wait time vs compute time?

Instrument the code with timing around IO calls and compute sections. JDK Flight Recorder (JFR) can measure time spent in different thread states automatically.

package com.bank.diagnostics;

public class TimingExample {
    public void processStatement(String accountId) {
        long ioStart = System.nanoTime();
        var data = accountRepo.findById(accountId);  // IO: wait time
        long ioEnd = System.nanoTime();

        long computeStart = System.nanoTime();
        var statement = generateStatement(data);      // CPU: compute time
        long computeEnd = System.nanoTime();

        long waitMs = (ioEnd - ioStart) / 1_000_000;
        long computeMs = (computeEnd - computeStart) / 1_000_000;
        // Log or export to metrics: waitMs=90, computeMs=10 → ratio=9
    }
}

What about virtual threads?

Java 21 virtual threads eliminate the sizing problem for IO-bound work. Virtual threads are lightweight (approximately 1 KB stack initially) and blocking is effectively free because the JVM unmounts blocked virtual threads from carrier threads. You can create millions of virtual threads without worrying about stack memory or context switching. The pool sizing formulas apply only to platform threads.

Should the pool size be configurable?

Yes. Externalize pool sizes to configuration files or environment variables so they can be tuned without redeployment. The formulas give starting points, but production load patterns may differ from estimates.

What are common interview follow-up questions?

• How do you validate pool sizing? Set the size from the formula, run a realistic load test (not micro-benchmark), watch CPU utilization and throughput. If CPU is under 70% and throughput plateaus, the pool may be undersized. If CPU is at 100% but throughput drops, the pool is oversized.
• What is the CallerRunsPolicy? When the queue is full, the submitting thread runs the task itself. This provides natural backpressure by slowing down the producer when the system is overloaded.