How do you debug an OutOfMemoryError?

Debugging steps:

1. Capture: -XX:+HeapDumpOnOutOfMemoryError auto-captures a dump.
2. Analyze: Open in Eclipse MAT. Look at Dominator Tree (biggest retained objects).
3. Find the leak: Trace the shortest path from GC root to the large object.
4. Fix: Remove the unintended reference, add cache eviction, close resources.

Enable automatic heap dump

The first step in debugging any OOM is ensuring you have evidence. Enable automatic heap dump capture so that when the JVM crashes, it writes a full snapshot of the heap.

java -XX:+HeapDumpOnOutOfMemoryError \
     -XX:HeapDumpPath=/tmp/heapdump.hprof \
     -Xmx2g -jar bank-app.jar

Without this flag, the JVM crashes and the evidence is gone. You never want to debug a production OOM without a heap dump.

OOM types and causes

The error message tells you which memory region ran out. Each type has different causes and fixes.

| Error | Cause | Fix |
|---|---|---|
| Java heap space | Heap full of live objects | Fix leak, increase -Xmx, or paginate large queries |
| Metaspace | Too many loaded classes | Fix classloader leak, increase MaxMetaspaceSize |
| GC overhead limit exceeded | GC running >98% time, freeing <2% | Fix leak, increase heap |
| Unable to create native thread | Too many threads or -Xss too large | Use bounded thread pool, reduce -Xss |

Analyze with Eclipse MAT

Eclipse Memory Analyzer (MAT) is the standard tool for analyzing heap dumps. It reads the .hprof file and provides several views to find the problem.

  1. Open heapdump.hprof in Eclipse MAT
  2. Run "Leak Suspects" report → MAT's automatic analysis
  3. Open Dominator Tree → shows objects retaining the most memory
  4. Right-click suspicious object → Path to GC Root (exclude weak refs)
  5. The path shows exactly what reference chain keeps the object alive
  6. Fix that reference in code

The Dominator Tree is the most powerful view. It ranks objects by retained heap size (how much memory would be freed if that object were garbage collected). The top entries are your prime suspects.

Common leak patterns in banking code

package com.bank.service;

import java.util.HashMap;
import java.util.Map;

public class TransactionCache {
    // LEAK: static map that only grows – nothing ever removes entries
    private static final Map<String, byte[]> cache = new HashMap<>();

    public void process(String txnId, byte[] data) {
        cache.put(txnId, data);  // never evicted → grows forever
    }
}
// FIX: use a bounded cache with eviction
// Caffeine.newBuilder().maximumSize(10_000).expireAfterWrite(Duration.ofHours(1)).build();

package com.bank.report;

import java.io.FileInputStream;
import java.io.InputStream;

public class ReportReader {
    // LEAK: unclosed stream holds OS resources and buffer memory
    public void read() {
        InputStream is = new FileInputStream("transactions.csv");  // never closed
    }

    // FIX: try-with-resources ensures close even on exception
    public void readCorrectly() throws Exception {
        try (InputStream is = new FileInputStream("transactions.csv")) {
            // process the file
        }  // auto-closed here
    }
}

Manual heap dump from a running process

Sometimes you want to take a dump before an OOM, for example when you notice the heap trending upward.

# Using jmap
jmap -dump:live,format=b,file=/tmp/dump.hprof <pid>

# Using jcmd (modern, preferred)
jcmd <pid> GC.heap_dump /tmp/dump.hprof

# Using JFR (can trigger from a running recording)
jcmd <pid> JFR.dump filename=/tmp/recording.jfr

What should a senior engineer know?

• In Kubernetes, use ephemeral containers or kubectl cp to retrieve the dump file. Set the dump path to a persistent volume so it survives pod restart.
• GC logs help distinguish a leak (heap grows over time) from under-sizing (heap is just too small for the workload).
• Never call System.gc() in production to "prevent" OOM. The JVM treats it as a suggestion, not a command, and it forces a Full GC that hurts performance.
• Weak references and SoftReferences are not a substitute for proper cache eviction. SoftReferences are cleared at the JVM's discretion, making cache behavior unpredictable.
• At National Bank of India, Rajesh configures every production service with heap dump on OOM, GC logging, and a dashboard alert when post-GC heap exceeds 80% of max. This catches leaks before they become customer-visible outages.