Definition
- jstat – lightweight CLI tool that polls GC and generation statistics from a running JVM without a heavy agent.
- jcmd – modern multi-purpose diagnostic entry point for heap info, thread dumps, heap dumps, native memory, and effective flags.
- jmap – takes heap dumps and histograms; in production prefer automatic dump-on-OOM or jcmd.
- Eclipse MAT – offline forensic tool for heap dumps; computes retained sizes, dominator tree, and leak-suspect reports.
- async-profiler – low-overhead sampling profiler producing CPU, allocation, and lock flame graphs safe for brief production use.
- Micrometer + Prometheus + Grafana – metrics facade exposing JVM metrics, scraped by Prometheus, visualized and alerted in Grafana.
- APM – distributed tracing that ties slow requests to specific services, methods, and GC pauses.
Monitoring the JVM is like running National Bank of India's control room. jstat and jcmd are the live gauges on the wall. jmap takes a full photograph of the vault contents when something looks wrong. Eclipse MAT is the forensic analyst studying that photograph. async-profiler is a heat-map showing where staff spend their time. Micrometer-Prometheus-Grafana is the 24/7 dashboard with alarms set on the few numbers that actually predict trouble.
Understanding the Concepts
What is jstat and how do you use it?
jstat is a lightweight command-line tool that polls GC and generation statistics from a running JVM without attaching a heavy agent. It is the first thing to run when you suspect GC trouble.
# Poll GC utilization every second
jstat -gcutil <pid> 1000
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 45.23 67.12 82.45 94.12 91.34 142 1.234 3 0.456 1.690
# S0/S1 = survivor spaces, E = Eden, O = Old Gen, M = Metaspace
# YGC/FGC = young/full GC count, YGCT/FGCT = cumulative time in seconds
If Old Gen (O column) stays above 80% even after full GCs, the live set is too large or growing. If FGC count keeps increasing, the JVM is thrashing.
What is jcmd and why is it the modern Swiss-army tool?
jcmd has largely superseded older single-purpose tools. It is one command for many diagnostic tasks.
# Show heap usage
jcmd <pid> GC.heap_info
# Take a heap dump
jcmd <pid> GC.heap_dump /var/log/bank/heap.hprof
# Print thread stacks
jcmd <pid> Thread.print
# Show effective JVM flags
jcmd <pid> VM.flags
# Show native memory (requires -XX:NativeMemoryTracking=summary)
jcmd <pid> VM.native_memory summary
package com.bank.debug;
import java.lang.management.ManagementFactory;
public class DiagnosticHelper {
public static void printPid() {
String name = ManagementFactory.getRuntimeMXBean().getName();
String pid = name.split("@")[0];
System.out.println("PID: " + pid);
System.out.println("Use: jcmd " + pid + " GC.heap_info");
}
}
How do you capture and analyze heap dumps?
jmap historically took heap dumps, but in production prefer the automatic dump-on-OOM flag or jcmd, because a manual jmap on a large live heap can pause the application.
# Automatic dump on OOM (preferred)
java -XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/var/log/bank/heapdumps/ \
-jar bank-transfer-service.jar
# Manual dump with jcmd (safer than jmap)
jcmd <pid> GC.heap_dump /var/log/bank/heap.hprof
# Manual dump with jmap (legacy)
jmap -dump:live,format=b,file=heap.hprof <pid>
Eclipse Memory Analyzer (MAT) is the offline forensic tool for heap dumps. It computes retained sizes, finds the biggest memory consumers, runs leak-suspect reports, and shows the dominator tree.
# MAT analysis workflow:
1. Open heap.hprof in MAT
2. Run "Leak Suspects" report
3. Check the Dominator Tree -- which object retains the most memory?
4. Example finding:
com.bank.cache.TransactionCache
retained: 1.2 GB
problem: unbounded HashMap holding every processed transaction
fix: add eviction policy (LRU with max size)
How does async-profiler work?
async-profiler is a low-overhead sampling profiler that produces CPU, allocation, and lock flame graphs with minimal impact, safe enough to run briefly in production.
# CPU flame graph for 30 seconds
./asprof -d 30 -f /tmp/bank-cpu.html <pid>
# Allocation flame graph -- shows where objects are allocated
./asprof -e alloc -d 30 -f /tmp/bank-alloc.html <pid>
# Lock contention flame graph
./asprof -e lock -d 30 -f /tmp/bank-lock.html <pid>
# Reading the flame graph:
# Wide bars = hot methods consuming the most CPU/allocations
# Example finding:
# com.bank.service.TransferService.validateAccount (42% CPU)
# Root cause: N+1 database query inside a loop
How do you set up Micrometer + Prometheus + Grafana?
Micrometer is the metrics facade that Spring Boot uses to expose JVM metrics. Prometheus scrapes them. Grafana visualizes and alerts.
<!-- pom.xml -- add Micrometer Prometheus registry --syn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>io.micrometersyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>micrometer-registry-prometheussyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
# application.yml -- expose Prometheus endpoint
management:
endpoints:
web:
exposure:
include: prometheus,health
metrics:
tags:
application: bank-transfer-service
# Verify metrics endpoint
curl http://localhost:8080/actuator/prometheus | grep jvm_memory
# Sample Prometheus metrics:
jvm_memory_used_bytes{area="heap",id="G1 Old Gen"} 524288000
jvm_memory_max_bytes{area="heap",id="G1 Old Gen"} 1073741824
jvm_gc_pause_seconds_count{action="end of minor GC"} 142
jvm_gc_pause_seconds_sum{action="end of minor GC"} 1.234
Which metrics should you alert on and why?
The metrics worth alerting on are few and chosen because they predict real trouble.
| Metric | Alert Threshold | Why |
|--------|----------------|-----|
| Old Gen % after full GC | > 80% sustained | Live set is large or growing |
| GC pause p99 | > 500ms (G1) | Worst-case stalls break latency SLOs |
| GC time ratio | > 10% of wall-clock time | JVM is thrashing in collection |
| Monotonic heap growth | Floor after full GC trending up | Classic memory leak signature |
# Grafana alert rule (Prometheus query)
# Monotonic heap growth -- the most valuable leak alarm
- alert: HeapLeakDetected
expr: |
increase(jvm_memory_used_after_gc_bytes{pool="G1 Old Gen"}[1h]) > 0
and jvm_memory_used_after_gc_bytes{pool="G1 Old Gen"}
/ jvm_memory_max_bytes{pool="G1 Old Gen"} > 0.7
for: 30m
labels:
severity: warning
annotations:
summary: "Heap floor rising after GC -- possible memory leak"
The key subtlety is to watch the floor after full GC, not the instantaneous usage. Healthy heaps are supposed to fill and be collected; it is the floor that must not keep rising.
What is the difference between live inspection and continuous monitoring?
| Aspect | Live Inspection (jstat, jcmd) | Continuous Monitoring (Micrometer+Prometheus) |
|--------|------------------------------|----------------------------------------------|
| When used | During incidents, debugging | Always running in production |
| Overhead | Near zero, on demand | Low, scrape every 15-30 seconds |
| History | None, real-time only | Full time-series with retention |
| Alerting | Manual observation | Automated alerts with thresholds |
| Setup | Built into JDK, no config | Requires Micrometer + Prometheus + Grafana |
# Incident workflow: alert fires, then use live tools to dig deeper
# Step 1: Grafana alert fires -- old gen > 80% after full GC
# Step 2: SSH into pod, run jstat to confirm
jstat -gcutil <pid> 1000
# Step 3: Take heap dump with jcmd
jcmd <pid> GC.heap_dump /tmp/heap.hprof
# Step 4: Download and analyze with Eclipse MAT
What are common mistakes in JVM monitoring?
# WRONG: alerting on instantaneous heap usage
# Healthy heaps fill up between GC cycles -- this fires constantly
alert: jvm_memory_used_bytes{area="heap"} > 800000000
# Result: alert fatigue, team ignores real problems
# CORRECT: alert on heap AFTER full GC (the floor)
alert: jvm_memory_used_after_gc_bytes{pool="G1 Old Gen"} > threshold
# This fires only when the live set is genuinely too large
# WRONG: running jmap -dump on a large production heap
jmap -dump:live,format=b,file=heap.hprof <pid>
# This can pause the application for seconds on a large heap
# CORRECT: let the automatic flag capture the dump on OOM
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/bank/ ...
# Or use jcmd which is generally safer
jcmd <pid> GC.heap_dump /tmp/heap.hprof
What should a senior engineer know about JVM monitoring?
A senior engineer thinks in three layers: live inspection with lightweight CLI tools for immediate diagnosis, deep offline analysis with MAT and async-profiler for root-cause investigation, and continuous metrics pipelines with Micrometer-Prometheus-Grafana for automated alerting. The skill is knowing which layer to reach for and when, and setting alerts on the few leading indicators that predict real trouble rather than flooding dashboards with noise.
Visual Overview
JVM Monitoring Three-Layer Architecture
=========================================
┌─────────────────────────────────────────────────────┐
│ Layer 3: Continuous Monitoring (always running) │
│ Micrometer ──▶ Prometheus ──▶ Grafana + Alerts │
│ APM ──▶ Distributed Tracing │
│ Alerts: old-gen%, GC pause p99, GC ratio, growth │
├─────────────────────────────────────────────────────┤
│ Layer 2: Deep Offline Analysis (on demand) │
│ Eclipse MAT ──▶ Dominator tree, leak suspects │
│ async-profiler ──▶ CPU / allocation flame graphs │
│ VisualVM ──▶ Interactive inspection (staging) │
├─────────────────────────────────────────────────────┤
│ Layer 1: Live Inspection (during incidents) │
│ jstat ──▶ GC utilization per second │
│ jcmd ──▶ heap info, thread dump, native memory │
│ jmap ──▶ heap dump (prefer jcmd or auto-dump) │
└─────────────────────────────────────────────────────┘
Alert Decision: What to Watch
==============================
After Full GC
│
▼
Old Gen floor rising? ──Yes──▶ ALERT: Memory leak
│
No
│
▼
GC pause p99 > SLO? ──Yes──▶ ALERT: Latency risk
│
No
│
▼
GC time > 10%? ──Yes──▶ ALERT: GC thrashing
│
No
│
▼
Healthy -- no action needed
Key Takeaway
JVM monitoring works in three layers. The first is live inspection with lightweight CLI tools: jstat for per-second GC utilization, jcmd as the modern multi-purpose diagnostic for heap info, thread dumps, native memory, and effective flags. The second is deep offline analysis: Eclipse MAT for heap dumps with the dominator tree and leak-suspect reports, and async-profiler for low-overhead CPU and allocation flame graphs. The third is continuous metrics: Micrometer exposing JVM metrics to Prometheus, Grafana dashboards with automated alerts, and APM for distributed tracing.
The metrics worth alerting on are deliberately few. Alert on old-gen occupancy that stays high after full collection, GC pause p99 that breaks latency SLOs, GC-time ratio that shows the JVM is thrashing, and most importantly monotonic heap growth where the floor after full GC trends upward without bound. The key subtlety is to watch the floor after full GC, not the instantaneous usage, because healthy heaps fill and get collected.
At National Bank of India a payments service slowed down over several days until someone restarted it. We added monotonic-growth and old-gen-after-GC alerts to Grafana, grabbed a heap dump with jcmd when the floor started climbing, followed the dominator tree in MAT, and found an unbounded cache holding every processed transaction.
Remember one sentence: alert on the heap floor after full GC, not on instantaneous usage, because the floor rising is the classic leak signature.
---