How to troubleshoot thread contention?

• Thread dump – a snapshot of all threads' states, stack traces, and lock information. Collected with jstack , kill -3 , or ThreadMXBean.dumpAllThreads().
• Thread states – many BLOCKED = lock contention. Many WAITING = pool workers idle or waiting for downstream. Many TIMED_WAITING = threads sleeping or on timeout.
• Lock identification – ThreadInfo.getLockName() shows which lock threads are blocked on. getLockOwnerName() shows the thread holding the lock.
• Root causes – (1) excessive locking: coarse lock or slow operation inside synchronized, (2) undersized pool: all workers busy, queue growing, (3) slow downstream: threads blocked on HTTP/DB calls, (4) connection pool contention: all connections in use, threads waiting for free connection.
• JFR – JDK Flight Recorder provides continuous recording with low overhead. Lock contention events, thread park durations, and hot methods are all captured.

How do you troubleshoot thread contention?

The systematic approach has four steps: collect evidence, analyze thread states, identify the bottleneck, and fix the root cause.

Step 1: Collect a thread dump

Take multiple dumps (3-5) spaced 5 seconds apart. A single dump is a snapshot; multiple dumps show trends and distinguish genuinely stuck threads from normal busy work.

jstack -l <PID> > dump1.txt
sleep 5
jstack -l <PID> > dump2.txt
sleep 5
jstack -l <PID> > dump3.txt

Or programmatically from inside the application:

package com.bank.diagnostics;

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;

public class ThreadDumpCollector {
    public static void dumpThreads() {
        ThreadMXBean tmx = ManagementFactory.getThreadMXBean();
        ThreadInfo[] infos = tmx.dumpAllThreads(true, true);
        for (ThreadInfo info : infos) {
            System.out.println(info.getThreadName() + ": " + info.getThreadState());
            if (info.getLockName() != null)
                System.out.println("  Blocked on: " + info.getLockName());
            if (info.getLockOwnerName() != null)
                System.out.println("  Lock held by: " + info.getLockOwnerName());
        }
    }
}

Step 2: Count thread states

Many BLOCKED threads on the same lock indicates lock contention. All pool workers RUNNABLE with a growing queue indicates an undersized pool. Many TIMED_WAITING in SocketInputStream.read indicates slow downstream services.

# Quick count of thread states
grep -c 'BLOCKED' dump.txt
grep -c 'WAITING' dump.txt
grep -c 'RUNNABLE' dump.txt
grep -c 'TIMED_WAITING' dump.txt

Step 3: Identify the lock owner

The thread dump shows which lock each BLOCKED thread is waiting for and which thread currently holds that lock. The lock owner's stack trace reveals what the holder is doing while other threads wait.

Step 4: Fix the root cause

Four common root causes with different fixes:

| Root cause | Thread dump signature | Fix |
|---|---|---|
| Excessive locking | 150 BLOCKED on one lock, owner in DB query | Move IO outside synchronized block |
| Undersized pool | All workers RUNNABLE, queue 5000+ deep | Increase pool size or optimize task duration |
| Slow downstream | 100 TIMED_WAITING in SocketInputStream | Add timeouts (5s), add circuit breaker |
| Connection pool | 100 WAITING in HikariPool.getConnection | Increase maxPoolSize, optimize queries |

How do you fix excessive locking in a banking transfer service?

The most common production fix is moving IO operations outside the synchronized block to minimize lock hold time.

package com.bank.service;

import java.math.BigDecimal;

public class TransferService {

    // WRONG: DB query inside synchronized block
    // Lock hold time: 2 seconds (DB query time)
    public void transferWrong(String fromId, String toId, BigDecimal amount) {
        synchronized (this) {
            BigDecimal fromBalance = accountRepo.getBalance(fromId);  // 2s DB query!
            BigDecimal toBalance = accountRepo.getBalance(toId);      // 2s DB query!
            accountRepo.updateBalance(fromId, fromBalance.subtract(amount));
            accountRepo.updateBalance(toId, toBalance.add(amount));
        }
    }

    // CORRECT: only synchronize the critical section
    // Lock hold time: microseconds (in-memory check + update)
    public void transferCorrect(String fromId, String toId, BigDecimal amount) {
        BigDecimal fromBalance = accountRepo.getBalance(fromId);  // outside lock
        BigDecimal toBalance = accountRepo.getBalance(toId);      // outside lock
        synchronized (this) {
            // Only synchronize the state mutation
            accountRepo.updateBalance(fromId, fromBalance.subtract(amount));
            accountRepo.updateBalance(toId, toBalance.add(amount));
        }
    }
}

What is JFR and how does it help?

JDK Flight Recorder provides continuous recording with low overhead (typically under 2% CPU). It captures lock contention events, thread park durations, hot methods, and GC pauses. Analyze recordings with JDK Mission Control.

# Start a JFR recording for 60 seconds
jcmd <PID> JFR.start duration=60s filename=contention.jfr
# Open in JDK Mission Control: "Lock Instances" tab shows most contended locks

What are common interview follow-up questions?

• How often should you take thread dumps? Take 3-5 dumps 5 seconds apart. A single dump is a snapshot; multiple dumps show trends and distinguish stuck threads from momentarily busy ones.
• What is the difference between BLOCKED and WAITING? BLOCKED means waiting for a synchronized monitor. WAITING means parked by LockSupport, Object.wait(), or Thread.join(). ReentrantLock contention shows as WAITING, not BLOCKED.
• How do you detect ReentrantLock contention? Use JFR jdk.ThreadPark events or async-profiler -e lock mode, since ReentrantLock waiters show as WAITING (parking), not BLOCKED.