Definition
- Evidence first – Collect CPU metrics per instance, separate application CPU from GC CPU, and run an async or sampling profiler to produce a flame graph before touching any code.
- GC split – If a large fraction of CPU is in garbage collection with the heap filling repeatedly, the cause is allocation pressure, not business logic.
- Flame graph – The profiler samples thread stacks at high frequency and names the hottest methods as the widest bars, removing all guesswork.
- Common causes – Tight loops or bad algorithms, expensive JSON or object serialization, lock contention showing as spinning threads, pathological regex backtracking, and GC pressure from excessive short-lived allocation.
- Fix the proven cause – Each cause has a distinct fingerprint and a specific fix; scaling out without diagnosis just multiplies a defect across more machines.
Imagine one teller at National Bank of India is sweating and working furiously non-stop, yet the queue barely moves. A reckless manager would just hire more tellers. A systematic manager watches exactly what that teller is doing over and over: is he re-counting the same stack of notes in an endless loop, is he laboriously re-writing a huge form, is he fighting with a colleague over a single shared ledger, or is he constantly cleaning his desk because he runs out of space (the garbage-collection equivalent). You observe the busy worker, find the one wasteful task, and fix that specific task.
Understanding the Concepts
What is sustained high CPU in a Java service?
Sustained high CPU means one or more JVM instances are pinning their CPU at or near 100 percent for an extended period, even when the incoming request rate does not justify it. In a banking system like National Bank of India, this might be the fraud-detection service or the transaction-processing service burning cores without producing proportional throughput. The key word is sustained: a brief spike during peak is normal, but minutes or hours of maxed CPU with no corresponding traffic increase signals a defect.
package com.bank.monitoring;
import java.lang.management.ManagementFactory;
import com.sun.management.OperatingSystemMXBean;
public class CpuMetricsCollector {
public static double getProcessCpuLoad() {
OperatingSystemMXBean osBean =
(OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
return osBean.getProcessCpuLoad() * 100; // percentage
}
public static void main(String[] args) throws InterruptedException {
for (int i = 0; i < 10; i++) {
System.out.printf("Process CPU: %.1f%%%n", getProcessCpuLoad());
Thread.sleep(2000);
}
}
}
Process CPU: 87.3%
Process CPU: 92.1%
Process CPU: 91.8%
Why must you collect evidence before changing anything?
High CPU has several distinct causes whose fixes are mutually exclusive. Adding servers to a tight-loop defect just burns more money. Optimizing business logic when the CPU is actually in GC is wasted effort. The flame graph routinely surprises even experienced engineers because the intuitive culprit is often wrong. Collecting evidence means you prove which threads and which methods are responsible before you invest time in a fix.
package com.bank.debug;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
public class ThreadDumpCollector {
public static void captureThreadDump() {
ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
ThreadInfo[] infos = threadBean.dumpAllThreads(true, true);
for (ThreadInfo info : infos) {
if (info.getThreadState() == Thread.State.RUNNABLE) {
System.out.println("RUNNABLE: " + info.getThreadName());
for (StackTraceElement element : info.getStackTrace()) {
System.out.println(" at " + element);
}
}
}
}
public static void main(String[] args) {
System.out.println("--- Thread Dump (RUNNABLE only) ---");
captureThreadDump();
}
}
--- Thread Dump (RUNNABLE only) ---
RUNNABLE: http-nio-8080-exec-3
at com.bank.fraud.NarrationValidator.validate(NarrationValidator.java:47)
at java.util.regex.Matcher.match(Matcher.java:1270)
How do you separate application CPU from GC CPU?
The JVM can burn CPU either running your code or collecting garbage, and the fixes are opposite. If GC logs show frequent or long collections with the heap repeatedly filling, the cause is GC pressure from excessive allocation, and you fix allocation and heap sizing, not business logic. If GC is quiet, the CPU is genuinely in your code.
package com.bank.debug;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.util.List;
public class GcCpuChecker {
public static void printGcStats() {
List<GarbageCollectorMXBean> gcBeans =
ManagementFactory.getGarbageCollectorMXBeans();
for (GarbageCollectorMXBean gc : gcBeans) {
System.out.printf("GC [%s]: count=%d, totalTime=%dms%n",
gc.getName(), gc.getCollectionCount(), gc.getCollectionTime());
}
}
public static void main(String[] args) {
printGcStats();
// If totalTime is a large fraction of uptime, GC is the CPU hog
}
}
GC [G1 Young Generation]: count=4820, totalTime=38200ms
GC [G1 Old Generation]: count=12, totalTime=980ms
If the young generation has consumed 38 seconds of CPU in just a few minutes, GC pressure is the likely cause.
How does a sampling profiler produce a flame graph?
An async or sampling profiler like async-profiler or Java Flight Recorder samples thread stacks cheaply at high frequency and aggregates them into a flame graph. The widest bars are exactly where CPU time is spent. It is low overhead and safe to run in production for a short window.
# Run async-profiler on a live JVM for 30 seconds
./profiler.sh -d 30 -f /tmp/flamegraph.html <PID>
# The output HTML shows a flame graph where
# the widest bar might be:
# com.bank.fraud.NarrationValidator.validate (72% of CPU)
# java.util.regex.Matcher.match (71% of CPU)
The flame graph names the hottest method directly, removing guesswork.
What does a tight loop or bad algorithm look like?
A tight loop is a method doing far more work than needed, for example an accidental O(n-squared) scan over a growing list. The flame graph shows one business method dominating.
package com.bank.service;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
public class TransactionMatcher {
// WRONG: O(n^2) nested loop on every call
public List<String> findDuplicates(List<String> referenceNumbers) {
List<String> duplicates = new ArrayList<>();
for (int i = 0; i < referenceNumbers.size(); i++) {
for (int j = i + 1; j < referenceNumbers.size(); j++) {
if (referenceNumbers.get(i).equals(referenceNumbers.get(j))) {
duplicates.add(referenceNumbers.get(i));
}
}
}
return duplicates;
}
}
package com.bank.service;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
public class TransactionMatcherFixed {
// CORRECT: O(n) using a Set
public Set<String> findDuplicates(List<String> referenceNumbers) {
Set<String> seen = new HashSet<>();
return referenceNumbers.stream()
.filter(ref -> !seen.add(ref))
.collect(Collectors.toSet());
}
}
What does expensive serialization look like?
Converting large objects to and from JSON on every request is CPU-heavy. The flame graph shows serialization library frames dominating. Fixes include serializing less data, reusing object mappers, and caching serialized forms.
package com.bank.service;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.math.BigDecimal;
public class StatementSerializer {
// WRONG: creating a new ObjectMapper per call
public String serializeStatement(Object statement) throws Exception {
ObjectMapper mapper = new ObjectMapper(); // expensive to create
return mapper.writeValueAsString(statement);
}
}
package com.bank.service;
import com.fasterxml.jackson.databind.ObjectMapper;
public class StatementSerializerFixed {
// CORRECT: reuse a single ObjectMapper instance (thread-safe for reads)
private static final ObjectMapper MAPPER = new ObjectMapper();
public String serializeStatement(Object statement) throws Exception {
return MAPPER.writeValueAsString(statement);
}
}
What does lock contention look like?
Lock contention shows up when many threads fight for the same lock. A thread dump shows many threads BLOCKED on the same monitor while one holds it.
package com.bank.service;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.Map;
public class AccountBalanceCache {
private final Map<String, BigDecimal> balances = new HashMap<>();
// WRONG: coarse-grained lock blocks ALL accounts on every read
public synchronized BigDecimal getBalance(String accountId) {
return balances.getOrDefault(accountId, BigDecimal.ZERO);
}
public synchronized void updateBalance(String accountId, BigDecimal amount) {
balances.put(accountId, amount);
}
}
package com.bank.service;
import java.math.BigDecimal;
import java.util.concurrent.ConcurrentHashMap;
public class AccountBalanceCacheFixed {
// CORRECT: lock-free concurrent map, no contention
private final ConcurrentHashMap<String, BigDecimal> balances =
new ConcurrentHashMap<>();
public BigDecimal getBalance(String accountId) {
return balances.getOrDefault(accountId, BigDecimal.ZERO);
}
public void updateBalance(String accountId, BigDecimal amount) {
balances.put(accountId, amount);
}
}
What does pathological regex backtracking look like?
A poorly written pattern against certain inputs explores an exponential number of match paths, pinning one thread at 100 percent CPU on a single request. The flame graph shows the regex matcher at the top.
package com.bank.fraud;
import java.util.regex.Pattern;
public class NarrationValidator {
// WRONG: nested quantifiers cause catastrophic backtracking
private static final Pattern BAD_PATTERN =
Pattern.compile("(a+)+b");
public boolean validate(String narration) {
// On input "aaaaaaaaaaaaaaaaaaaaaa" (no 'b'), this hangs
return BAD_PATTERN.matcher(narration).matches();
}
}
package com.bank.fraud;
import java.util.regex.Pattern;
public class NarrationValidatorFixed {
// CORRECT: possessive quantifier prevents backtracking
private static final Pattern SAFE_PATTERN =
Pattern.compile("a++b");
private static final int MAX_LENGTH = 500;
public boolean validate(String narration) {
if (narration.length() > MAX_LENGTH) {
return false; // bound input length as a safety net
}
return SAFE_PATTERN.matcher(narration).matches();
}
}
# BAD_PATTERN on "aaaaaaaaaaaaaaaaaaaaaa":
# hangs for minutes, CPU pinned at 100%
# SAFE_PATTERN on the same input:
# returns false in microseconds
What does GC pressure look like, and how do you fix it?
GC pressure means heavy short-lived allocation forces constant young-generation collection, or a leak forces repeated full collections. The signature is high GC CPU and allocation rate rather than business frames in the flame graph.
package com.bank.service;
import java.math.BigDecimal;
public class InterestCalculator {
// WRONG: creates a new BigDecimal wrapper on every iteration
public BigDecimal calculateCompound(BigDecimal principal, double rate, int months) {
BigDecimal result = principal;
for (int i = 0; i < months; i++) {
result = result.multiply(new BigDecimal(String.valueOf(1 + rate)));
}
return result;
}
}
package com.bank.service;
import java.math.BigDecimal;
public class InterestCalculatorFixed {
// CORRECT: reuse the multiplier, avoid per-iteration allocation
public BigDecimal calculateCompound(BigDecimal principal, double rate, int months) {
BigDecimal multiplier = BigDecimal.valueOf(1 + rate);
BigDecimal result = principal;
for (int i = 0; i < months; i++) {
result = result.multiply(multiplier);
}
return result;
}
}
What are common mistakes when debugging high CPU?
The most common mistake is scaling out before diagnosing the root cause. Adding servers to a tight-loop or regex-backtracking defect multiplies the problem across more machines at higher cost without solving anything.
package com.bank.debug;
// WRONG approach: jumping to horizontal scaling
public class WrongApproach {
// "CPU is high, let's add 3 more instances!"
// Each new instance runs the same bad regex, burns the same CPU,
// and the shared downstream DB now gets 4x the load.
}
package com.bank.debug;
// CORRECT approach: profile first, then fix the specific cause
public class CorrectApproach {
// 1. Confirm symptom with CPU metrics per instance
// 2. Check GC logs to split app CPU from GC CPU
// 3. Run async-profiler for 30 seconds
// 4. Read the flame graph to find the hottest method
// 5. Fix that specific method (regex, loop, serialization, lock)
// 6. Verify CPU dropped, then re-evaluate capacity
}
What should a senior engineer know?
A senior engineer knows that high CPU and high latency are different problems. A service can be slow while CPU is idle, which points at waiting on IO, locks, or dependencies, not a computing problem. The senior also captures evidence before restarting, because a restart hides the cause. Correlating the onset of high CPU with deploys, config changes, and input changes is critical, since a new regex or serialization path often arrives with a release.
package com.bank.debug;
import java.time.Instant;
import java.util.logging.Logger;
public class HighCpuRunbook {
private static final Logger LOG = Logger.getLogger(HighCpuRunbook.class.getName());
public void onHighCpuAlert(String serviceName, double cpuPercent) {
LOG.warning(String.format(
"[%s] High CPU alert for %s: %.1f%%. "
+ "Step 1: Capture thread dump and GC stats NOW before restart. "
+ "Step 2: Check last deploy timestamp. "
+ "Step 3: Run async-profiler for 30s.",
Instant.now(), serviceName, cpuPercent));
}
}
What are common interview follow-up questions?
Interviewers often follow up with: How would you diagnose if the CPU spike correlates with a specific deploy? How do you distinguish user CPU from system CPU on Linux? How would you handle high CPU in a containerized environment where profiler access is limited? What is the difference between a sampling profiler and an instrumenting profiler? How does JFR compare to async-profiler? These all test whether you understand the evidence-first methodology rather than guessing.
Visual Overview
┌─────────────────────────────────────────────────────────┐ │ HIGH CPU DEBUGGING FLOW │ ├─────────────────────────────────────────────────────────┤ │ │ │ 1. Confirm Symptom │ │ ┌──────────────────────┐ │ │ │ CPU Metrics / Instance│ │ │ │ All instances or one? │ │ │ │ Tracks traffic? │ │ │ └──────────┬───────────┘ │ │ │ │ │ 2. Split: App CPU vs GC CPU │ │ ┌──────────▼───────────┐ │ │ │ Check GC Logs │ │ │ └───┬──────────────┬───┘ │ │ │ │ │ │ GC Heavy GC Quiet │ │ ┌────▼────┐ ┌────▼────────────┐ │ │ │Fix heap │ │Profile app code │ │ │ │+ alloc │ │ │ │ │ └─────────┘ └────┬────────────┘ │ │ │ │ │ 3. Run Async Profiler │ │ │ ┌──────────────────▼──────────┐ │ │ │ Flame Graph: widest bar = │ │ │ │ hottest method │ │ │ └──────────────────┬──────────┘ │ │ │ │ │ 4. Match to Cause │ │ │ ┌──────────────────▼──────────────────────┐ │ │ │ Tight Loop │ Serialization │ Lock Spin │ │ │ │ Regex BT │ GC Pressure │ │ │ │ └──────────────────┬──────────────────────┘ │ │ │ │ │ 5. Fix Specific Cause │ │ │ ┌──────────────────▼──────────┐ │ │ │ Algorithm, mapper reuse, │ │ │ │ ConcurrentHashMap, regex │ │ │ │ rewrite, allocation reduce │ │ │ └─────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘
Key Takeaway
Debugging sustained high CPU in a Java service always starts with collecting evidence before touching any code. At National Bank of India, when the fraud-detection service pinned every core and stayed pinned even after traffic dropped, the team wanted to add instances. Pankaj insisted on profiling first. The first step was confirming whether GC was the culprit using GC logs, which it was not, so the CPU was genuinely in application code. Running async-profiler for 30 seconds produced a flame graph that put 72 percent of CPU inside the regex matcher used to validate transaction narrations.
The common causes each have a distinct fingerprint. A tight loop or bad algorithm shows one business method dominating the flame graph. Expensive serialization shows Jackson or mapper frames on top. Lock contention shows many threads BLOCKED on the same monitor in a thread dump. Pathological regex backtracking shows the regex matcher pinning a core on a single request. GC pressure shows high GC CPU and allocation rate rather than business frames. In Rajesh's case, it was classic catastrophic backtracking caused by a pattern with nested quantifiers hitting a crafted long narration input.
The fix was rewriting the regex to use a possessive quantifier and bounding the input length, which dropped the validation from seconds to microseconds and CPU returned to normal. The lesson Pankaj teaches every junior engineer is: never fix before you measure, prefer a sampling profiler over manual thread dumps, correlate the onset with the deploy that introduced the new code, and capture evidence before restarting because a restart hides the cause.
Remember one sentence: Debug high CPU by proving which threads and methods burn it with an async profiler, split app CPU from GC CPU, then fix the proven cause, which is almost always a tight loop, expensive serialization, lock contention, regex backtracking, or GC pressure.
---