Definition
- Cgroup memory limit – the hard memory cap enforced by Linux on a container; exceeding it triggers an immediate OOM-kill.
- Non-heap memory – everything outside the Java heap: Metaspace (class metadata), code cache (JIT output), thread stacks, direct byte buffers, GC bookkeeping, OS page cache.
- Concurrent allocation slack – the free memory ZGC needs to keep allocating into while it concurrently reclaims; G1 does not need as much because it pauses during evacuation.
- Colored pointer overhead – extra native memory from ZGC's multi-mapped virtual memory and page-table structures.
- MaxRAMPercentage – a JVM flag that sets the heap as a percentage of the detected container limit, replacing hardcoded -Xmx.
The bank rents a floor with a strict fire-safety occupancy limit (the pod memory limit). The main hall is the heap, but you also need corridors, stairwells, a staff room, and emptying space for the cleaning crew to shuffle furniture around (ZGC's concurrent work). If you cram the main hall to 90% of the floor's limit, there is no room for the crew to move desks during a clean-up, total occupancy crosses the fire limit, and the fire marshal (the cgroup OOM killer) evacuates the whole floor.
Understanding the Concepts
Why does the cgroup limit matter?
A container has a hard memory limit enforced by the Linux cgroup. If the process's resident set size (RSS) crosses that limit, the kernel OOM-killer terminates the process and Kubernetes records the pod as OOMKilled. The limit applies to total memory, not just the Java heap.
# Kubernetes pod with 4 GB memory limit
resources:
limits:
memory: "4Gi"
requests:
memory: "4Gi"
# Check container memory limit from inside the pod
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# 4294967296 (4 GB)
What makes up the JVM's total memory footprint?
The heap is only part of the total footprint. Non-heap memory adds significantly.
JVM memory breakdown for a typical banking service: Heap (-Xmx): 3.0 GB Metaspace (classes): 150 MB Code cache (JIT): 240 MB Thread stacks (200 x 1MB): 200 MB Direct buffers (Netty): 256 MB GC internal structures: 50 MB OS page cache: ~100 MB ────────────────────────────────── Total RSS: ~4.0 GB ← exceeds 4 GB limit!
# Check actual JVM memory breakdown with Native Memory Tracking
java -XX:NativeMemoryTracking=summary \
-XX:+UseZGC -Xmx3g \
-jar transfer-service.jar
# In another terminal:
jcmd $(pgrep -f transfer-service) VM.native_memory summary
Native Memory Tracking:
Total: reserved=5234MB, committed=3812MB
- Java Heap: reserved=3072MB, committed=3072MB
- Class (Metaspace): reserved=256MB, committed=152MB
- Thread: reserved=206MB, committed=206MB
- Code: reserved=250MB, committed=243MB
- GC: reserved=148MB, committed=48MB
- Internal: reserved=12MB, committed=12MB
- Other: reserved=1290MB, committed=79MB
Why does ZGC need more headroom than G1?
Two reasons: concurrent allocation slack and colored pointer native overhead.
G1: pauses during evacuation → heap is "frozen" during GC, no concurrent allocation needed → non-heap overhead is standard ZGC: reclaims WHILE application allocates → must keep free pages available for concurrent allocation → colored pointers use multi-mapped virtual memory → extra page tables and mapping structures consume native memory → total footprint is higher than G1 for the same heap size
# Same heap size, different total footprint:
# G1: -Xmx3g → total RSS ~3.5 GB
# ZGC: -Xmx3g → total RSS ~3.8 GB (concurrent + native overhead)
Why does 90% get the pod OOMKilled?
If the heap is set to 90% of the container limit, non-heap memory plus ZGC's concurrent allocation slack pushes total RSS over the limit during a GC cycle.
# WRONG: 90% of 4 GB = 3.6 GB heap
java -XX:+UseZGC -Xmx3600m -jar transfer-service.jar
# During a GC cycle:
# Heap: 3600 MB
# Non-heap: ~700 MB
# Total RSS: ~4300 MB > 4096 MB limit
# → OOMKilled!
Pod event:
Type: Warning
Reason: OOMKilled
Message: Container killed due to exceeding memory limit
Last State: OOMKilled (exit code 137)
How do you size ZGC correctly in a container?
Set the heap to 70-75% of the pod limit, reserving 25-30% for everything non-heap.
# CORRECT: MaxRAMPercentage for ZGC
java -XX:+UseZGC \
-XX:MaxRAMPercentage=72.0 \
-XX:+AlwaysPreTouch \
-XX:+ExitOnOutOfMemoryError \
-jar transfer-service.jar
# For G1, 75-80% is usually safe
java -XX:+UseG1GC \
-XX:MaxRAMPercentage=78.0 \
-jar transfer-service.jar
# Kubernetes deployment with correct JVM flags
spec:
containers:
- name: transfer-service
image: registry.nbi.com/transfer-service:2.5.0
resources:
limits:
memory: "4Gi"
requests:
memory: "4Gi"
env:
- name: JAVA_OPTS
value: >-
-XX:+UseZGC
-XX:MaxRAMPercentage=72.0
-XX:+AlwaysPreTouch
-XX:+ExitOnOutOfMemoryError
-Xlog:gc*:file=/var/log/gc.log
How do you verify headroom is sufficient?
Use Native Memory Tracking to check the actual footprint versus the container limit.
# Enable NMT for production debugging
java -XX:NativeMemoryTracking=summary \
-XX:+UseZGC -XX:MaxRAMPercentage=72.0 \
-jar transfer-service.jar
# Check memory usage
jcmd $(pgrep -f transfer-service) VM.native_memory summary
# Compare against container limit
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
What are common mistakes?
The most common mistake is carrying over a 90% heap ratio from bare metal into a container without accounting for the cgroup hard limit.
# WRONG: hardcoded -Xmx from bare metal days
java -XX:+UseZGC -Xmx3600m -jar app.jar
# Pod limit is 4 GB, but 3600m + non-heap = OOMKilled
# CORRECT: use MaxRAMPercentage so it tracks the limit
java -XX:+UseZGC -XX:MaxRAMPercentage=72.0 -jar app.jar
# If pod limit changes to 8 GB, heap automatically adjusts
What should a senior engineer know?
A senior engineer uses MaxRAMPercentage instead of hardcoded -Xmx, adds ExitOnOutOfMemoryError for clean fast-fail, AlwaysPreTouch for latency-sensitive services, and verifies the real footprint with Native Memory Tracking. At National Bank of India, the transfer API pods were being OOMKilled because someone carried over a 90% heap from bare metal; moving to MaxRAMPercentage of 72 with ExitOnOutOfMemoryError and AlwaysPreTouch stopped the kills.
package com.bank.monitoring;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
public class HeapMonitor {
public static void printHeapInfo() {
MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
long heapMax = memoryBean.getHeapMemoryUsage().getMax();
long heapUsed = memoryBean.getHeapMemoryUsage().getUsed();
long nonHeap = memoryBean.getNonHeapMemoryUsage().getUsed();
System.out.printf("Heap max: %d MB%n", heapMax / 1024 / 1024);
System.out.printf("Heap used: %d MB%n", heapUsed / 1024 / 1024);
System.out.printf("Non-heap used: %d MB%n", nonHeap / 1024 / 1024);
}
public static void main(String[] args) {
printHeapInfo();
}
}
Heap max: 2867 MB
Heap used: 1234 MB
Non-heap used: 187 MB
Visual Overview
Container memory limit: 4 GB ┌──────────────────────────────────────────────────┐ │ │ │ ┌──────────────────────────────┐ │ │ │ Java Heap │ ~72% = 2.9 GB│ │ │ (-Xmx) │ │ │ └──────────────────────────────┘ │ │ │ │ ┌────────┐┌──────┐┌───────┐┌─────────┐ │ │ │Metaspace││Thread││Code ││Direct │ ~28% │ │ │150 MB ││stacks││cache ││buffers │ │ │ │ ││200 MB││240 MB ││256 MB │ │ │ └────────┘└──────┘└───────┘└─────────┘ │ │ │ │ ZGC adds: concurrent allocation slack │ │ + colored pointer page tables │ └──────────────────────────────────────────────────┘ At 90% heap: no room → OOMKilled during GC At 72% heap: headroom for non-heap + concurrent work
Key Takeaway
In a container, the cgroup memory limit applies to the entire process footprint, not just the Java heap. Metaspace, code cache, thread stacks, direct buffers, and GC bookkeeping all share the same limit. ZGC needs more headroom than G1 because it allocates concurrently while reclaiming and its colored pointers add native page-table overhead.
The rule of thumb is to set the heap to 70-75% of the pod limit for ZGC, reserving the rest for non-heap memory and concurrent slack. Use MaxRAMPercentage so the heap tracks the container limit automatically. At 90%, the non-heap memory plus ZGC's concurrent work pushes total RSS over the limit during a GC cycle, and the pod is OOMKilled.
At National Bank of India, moving from a hardcoded 90% heap to MaxRAMPercentage of 72 with AlwaysPreTouch and ExitOnOutOfMemoryError eliminated the OOMKilled events on the transfer API pods.
Remember one sentence: the heap is not the whole footprint, so set -Xmx to 70-75% of the container limit for ZGC, because 90% gets you OOM-killed mid-GC.
---