What are the different OutOfMemoryError types, and what does each indicate? | Advanced Java

Q: What does Direct buffer memory mean?

Offheap direct ByteBuffers used by NIO and Netty exceeded the limit. These buffers are freed only when their Java wrapper is garbagecollected and the cleaner runs. package com.bank.diagnostic; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.List; public class DirectBufferDem

Q: Which tool do you use for each error?

Enable Native Memory Tracking java XX:NativeMemoryTracking=summary ... Query it jcmd <pid VM.native_memory summary

Definition

Every OutOfMemoryError names the region that ran out, and knowing whether it is heap, Metaspace, direct memory, threads, or compressed class space tells you exactly where to look.

Java heap space – the object heap is full and GC cannot reclaim enough; caused by leaks or an undersized heap.
GC overhead limit exceeded – the JVM spends over 98% of time in GC while reclaiming almost nothing; effectively a thrashing heap.
Metaspace – the native region for class metadata is full; usually a classloader leak or heavy dynamic class generation.
Direct buffer memory – off-heap NIO direct buffers exceeded -XX:MaxDirectMemorySize; invisible to heap dumps.
unable to create native thread – the OS refused a new thread due to thread count, ulimit, or stack size limits.
Compressed class space – the capped region for compressed class pointers (64-bit JVMs) is full.

Think of National Bank of India's building running out of a specific kind of space. Java heap space is the main vault being full of cash boxes with nowhere to put a new one. GC overhead limit exceeded is the cleaning crew working around the clock but barely clearing any room. Metaspace is the filing room for rulebooks (class definitions) overflowing. Direct buffer memory is the external warehouse (off-heap) hitting its cap. Unable to create native thread is the HR office refusing to hire another teller. Compressed class space is a small shelf for short address labels filling up.

Understanding the Concepts

What does Java heap space mean?

The heap is where your objects live. When it is full and the garbage collector cannot free enough to satisfy an allocation, you get this error. The cause is usually a genuine leak (holding references you no longer need) or an undersized heap.

JAVA

package com.bank.diagnostic;

import java.util.ArrayList;
import java.util.List;
import java.math.BigDecimal;

public class HeapSpaceDemo {
    // WRONG: static list grows without bound
    private static final List<BigDecimal> statementCache = new ArrayList<>();

    public static void main(String[] args) {
        // Simulate leaking statement data
        for (int i = 0; i < Integer.MAX_VALUE; i++) {
            statementCache.add(new BigDecimal("1000.00"));
        }
    }
}

BASH

java -Xmx64m com.bank.diagnostic.HeapSpaceDemo

TEXT

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3210)
    at java.util.ArrayList.grow(ArrayList.java:265)
    at com.bank.diagnostic.HeapSpaceDemo.main(HeapSpaceDemo.java:12)

BASH

# Diagnose with a heap dump
jmap -dump:live,format=b,file=heap.hprof <pid>
# Or automatically:
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap.hprof ...

What does GC overhead limit exceeded mean?

The throughput collector throws this when it spends over 98% of time in GC while recovering less than 2% of the heap. It is effectively a thrashing heap.

JAVA

package com.bank.diagnostic;

import java.util.HashMap;
import java.util.Map;

public class GcOverheadDemo {
    public static void main(String[] args) {
        Map<Integer, byte[]> map = new HashMap<>();
        // Fill the heap so GC thrashes
        for (int i = 0; i < Integer.MAX_VALUE; i++) {
            map.put(i, new byte[1024]); // 1 KB per entry
        }
    }
}

BASH

java -Xmx32m com.bank.diagnostic.GcOverheadDemo

TEXT

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

The root cause and fix are the same as Java heap space: find the retained references with a heap dump.

What does Metaspace mean?

Metaspace stores class metadata in native memory (since Java 8, replacing PermGen). The usual cause is a classloader leak from repeated redeploys or heavy dynamic class generation.

JAVA

package com.bank.diagnostic;

import java.net.URL;
import java.net.URLClassLoader;

public class MetaspaceDemo {
    public static void main(String[] args) throws Exception {
        // Simulate classloader leak: load classes without releasing classloaders
        while (true) {
            URLClassLoader loader = new URLClassLoader(
                    new URL[]{new URL("file:///tmp/classes/")});
            loader.loadClass("com.bank.DynamicClass");
            // Leak: never close the loader, classes stay in Metaspace
        }
    }
}

BASH

java -XX:MaxMetaspaceSize=64m com.bank.diagnostic.MetaspaceDemo

TEXT

Exception in thread "main" java.lang.OutOfMemoryError: Metaspace

BASH

# Monitor loaded class count
jcmd <pid> VM.classloader_stats
# A steadily rising class count across redeploys = classloader leak

What does Direct buffer memory mean?

Off-heap direct ByteBuffers used by NIO and Netty exceeded the limit. These buffers are freed only when their Java wrapper is garbage-collected and the cleaner runs.

JAVA

package com.bank.diagnostic;

import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;

public class DirectBufferDemo {
    public static void main(String[] args) {
        List<ByteBuffer> buffers = new ArrayList<>();
        // Allocate direct buffers without releasing
        while (true) {
            buffers.add(ByteBuffer.allocateDirect(1024 * 1024)); // 1 MB
        }
    }
}

BASH

java -XX:MaxDirectMemorySize=64m com.bank.diagnostic.DirectBufferDemo

TEXT

Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory

A heap dump shows nothing for this error because the memory is off-heap. Use Native Memory Tracking instead.

What does unable to create native thread mean?

This is not about heap at all. The OS refused a new thread, usually because the process hit a thread or memory limit.

JAVA

package com.bank.diagnostic;

public class ThreadLeakDemo {
    public static void main(String[] args) {
        // Create threads without limit
        while (true) {
            new Thread(() -> {
                try {
                    Thread.sleep(Long.MAX_VALUE);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }).start();
        }
    }
}

TEXT

Exception in thread "main" java.lang.OutOfMemoryError:
    unable to create native thread: possibly out of memory or
    process/resource limits reached

BASH

# Fix: reduce threads, pool them, lower stack size, raise OS limits
java -Xss256k ...              # smaller stack per thread
ulimit -u 65535                # raise process limit

What does Compressed class space mean?

On 64-bit JVMs, compressed class pointers use a capped region. When it fills, raise CompressedClassSpaceSize or reduce loaded classes.

BASH

java -XX:CompressedClassSpaceSize=32m ...
# Error:
# java.lang.OutOfMemoryError: Compressed class space

# Fix:
java -XX:CompressedClassSpaceSize=256m ...

Which tool do you use for each error?

| Error | Region | Tool |
|---|---|---|
| Java heap space | Heap | Heap dump + Eclipse MAT |
| GC overhead limit | Heap | Heap dump + MAT |
| Metaspace | Native (class metadata) | jcmd classloader stats, class count |
| Direct buffer memory | Native (off-heap) | NMT, BufferPool MBean |
| unable to create native thread | OS | Thread dump, ulimit, pmap |
| Compressed class space | Native (class pointers) | jcmd, class count |

BASH

# Enable Native Memory Tracking
java -XX:NativeMemoryTracking=summary ...

# Query it
jcmd <pid> VM.native_memory summary

What should a senior engineer know?

The message is a map, not a diagnosis. Heap errors point to the Java heap and need a heap dump. Metaspace and compressed-class errors point to classloading and need class-count tracking. Direct memory and native-thread errors point to native memory and the OS, where a heap dump shows nothing.

JAVA

package com.bank.diagnostic;

import java.lang.management.BufferPoolMXBean;
import java.lang.management.ManagementFactory;

public class DirectBufferMonitor {
    public static void main(String[] args) {
        // Monitor direct buffer usage
        for (BufferPoolMXBean pool :
                ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class)) {
            if ("direct".equals(pool.getName())) {
                System.out.println("Direct buffers: " + pool.getCount());
                System.out.println("Memory used: " + pool.getMemoryUsed());
                System.out.println("Total capacity: " + pool.getTotalCapacity());
            }
        }
    }
}

TEXT

Direct buffers: 12
Memory used: 12582912
Total capacity: 12582912

Visual Overview

DIAGRAM

  OutOfMemoryError Decision Tree:

  OOM Message
  ├── "Java heap space"           → Heap dump → MAT → find leak
  ├── "GC overhead limit"         → Heap dump → MAT → same as above
  ├── "Metaspace"                 → Class count → classloader leak?
  ├── "Compressed class space"    → Same as Metaspace
  ├── "Direct buffer memory"      → NMT → BufferPool MBean → Netty?
  └── "unable to create native    → Thread dump → ulimit → pool threads
       thread"

  JVM Memory Regions:
  ┌─────────────────────────────────────────┐
  │  Process RSS                             │
  │                                          │
  │  ┌──────────────────┐  ┌──────────────┐ │
  │  │   Java Heap       │  │  Metaspace   │ │
  │  │  (heap space,     │  │ (class meta, │ │
  │  │   GC overhead)    │  │  compressed  │ │
  │  │                   │  │  class space) │ │
  │  └──────────────────┘  └──────────────┘ │
  │                                          │
  │  ┌──────────────────┐  ┌──────────────┐ │
  │  │ Direct Buffers    │  │ Thread Stacks│ │
  │  │ (off-heap NIO)    │  │ (native)     │ │
  │  └──────────────────┘  └──────────────┘ │
  └─────────────────────────────────────────┘

Key Takeaway

OutOfMemoryError is thrown when the JVM cannot satisfy a memory request, and the key insight is that the message names the exact region that ran out. Java heap space means the object heap is full, either from a leak or an undersized heap, and you diagnose it with a heap dump and Eclipse MAT. GC overhead limit exceeded is a related signal that the collector is thrashing, spending over 98% of time collecting while recovering almost nothing.

Metaspace means the native region for class metadata is full, usually from a classloader leak during repeated redeploys or heavy proxy generation, and you diagnose it by watching the loaded-class count rise. Direct buffer memory is off-heap: NIO and Netty allocate direct ByteBuffers outside the heap, and when they exceed MaxDirectMemorySize or are not released, a heap dump shows nothing because the memory is not on the heap, so you need Native Memory Tracking instead. Unable to create native thread is not about JVM memory at all; the OS refused a new thread because you hit a thread or process limit.

At National Bank of India Rajesh has seen each of these in production. The statement service died with Java heap space because a static list cached every transaction forever. A frequently redeployed service died with Metaspace from a classloader leak. A file-download path died with Direct buffer memory from un-freed NIO buffers that only NMT revealed. And an old thread-per-request design died with unable to create native thread under load, fixed by pooling.

Remember one sentence: the OutOfMemoryError message names the region that ran out, so heap errors need a heap dump, Metaspace errors need class-count tracking, and native-memory errors need NMT and the OS view.

---