Definition
OOMKilled means the Linux kernel's out-of-memory killer terminated a container because its cgroup memory usage exceeded the limit.
- Cgroup enforcement – the container memory limit caps the entire resident memory of the process, not just the Java heap.
- Off-heap memory – includes thread stacks, metaspace, JIT code cache, direct byte buffers (Netty, NIO), and native allocations.
- Dashboard blind spot – most dashboards graph only heap usage, so the heap looks fine while total memory crosses the limit.
- Container-aware JVM – modern JVMs honor the cgroup limit; set
-XX:MaxRAMPercentage=75to leave 25% for off-heap. - Right-sizing – request is the typical working set (for scheduling), limit is above the real peak (hard cap).
At the National Bank of India, imagine a teller's cubicle has a strict space limit for everything they bring in. Your dashboard only measures the papers on the desk (JVM heap), but the cubicle limit counts the desk papers plus the filing cabinet, the chair, the coat rack, and the coffee machine (off-heap and native memory). So the cubicle can overflow and security evicts the teller even though the desk papers looked modest.
Understanding the Concepts
What does OOMKilled mean?
OOMKilled means the Linux kernel's out-of-memory killer sent SIGKILL (signal 9) to a container because its cgroup memory usage crossed the container's memory limit. The exit code is 137 (128 + 9), and kubectl describe pod shows Reason: OOMKilled.
kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137
# Started: 2025-01-15T10:00:00Z
# Finished: 2025-01-15T10:45:00Z
kubectl get pods -n prod
# NAME READY STATUS RESTARTS
# transfer-service-6d4f9c7b8a-x9k 0/1 OOMKilled 3
Why does monitoring look normal during an OOM kill?
Most dashboards graph only the Java heap usage. But the container limit caps the entire resident memory of the JVM process. The heap is only one region. Off-heap and native memory also count toward the cgroup limit.
Off-heap memory includes:
- Thread stacks each thread costs 0.5-1 MB; 200 threads = 100-200 MB.
- Metaspace for class metadata, grows with the number of loaded classes.
- JIT code cache for compiled native code, typically 240 MB default.
- Direct byte buffers used by Netty, NIO, and gRPC; can be large under load.
- Memory-mapped files and native allocations by libraries.
# Dashboard shows heap at 60% - looks fine
# But total container memory:
# Heap: 600 MB (60% of 1 GB max)
# Thread stacks: 150 MB (300 threads x 0.5 MB)
# Metaspace: 80 MB
# Code cache: 60 MB
# Direct buffers: 200 MB (Netty under load)
# ─────────────────────
# Total: 1,090 MB <- exceeds 1 GB limit!
# Container limit: 1,024 MB
# Result: OOMKilled while heap graph shows 60%
How do container-aware JVM flags work?
Older JVMs read the host machine's total memory, not the container limit. Inside a container with a 1 GB limit, an old JVM on a 32 GB node might size its default heap to 8 GB and get killed immediately. Modern JVMs (Java 10+) are container-aware by default.
# WRONG - hardcoded heap equal to the entire container limit
# Leaves nothing for off-heap -> guaranteed OOM
java -Xmx1024m -jar transfer-service.jar
# Container limit: 1024 MB, heap max: 1024 MB, off-heap: 0 MB available
# CORRECT - percentage-based, leaves room for off-heap
java -XX:MaxRAMPercentage=75.0 -jar transfer-service.jar
# Container limit: 1024 MB, heap max: ~768 MB, off-heap: ~256 MB available
# Verify the JVM sees the container limit
java -XX:+PrintFlagsFinal -version 2>&1 | grep MaxHeapSize
# MaxHeapSize = 805306368 (768 MB - 75% of 1024 MB)
# Deployment with container-aware JVM settings
apiVersion: apps/v1
kind: Deployment
metadata:
name: transfer-service
spec:
template:
spec:
containers:
- name: transfer
image: nbi-registry.example.com/transfer-service:2.0.0
resources:
requests:
memory: 768Mi # typical working set
limits:
memory: 1Gi # hard cap above real peak
env:
- name: JAVA_OPTS
value: "-XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport"
How do you right-size requests and limits?
Right-sizing is a measurement exercise, not a guess. The request is what the scheduler reserves and should reflect the typical working set. The limit is the hard cap and should sit above the observed real peak of total memory with headroom.
# Step 1: Measure actual container memory under realistic load
# Use container_memory_working_set_bytes, NOT just jvm_memory_used
kubectl top pods -n prod
# NAME CPU MEMORY
# transfer-service-6d4f9c7b8a-x9k 250m 720Mi <- total working set
# Step 2: Observe over time under peak load
# Prometheus query for the real peak:
# max_over_time(container_memory_working_set_bytes{pod=~"transfer.*"}[7d])
# Result: 850 Mi peak
# Step 3: Set request near steady state, limit above peak
# Request: 768 Mi (typical)
# Limit: 1024 Mi (above 850 Mi peak + headroom)
What are the QoS implications?
When request equals limit for both CPU and memory, the Pod is Guaranteed QoS and is the last to be evicted under node memory pressure. When request is below limit, the Pod is Burstable and more exposed to eviction.
# Guaranteed QoS - safest for critical banking services
resources:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "1"
memory: 1Gi
# Burstable QoS - more exposed to eviction
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
kubectl get pod transfer-service-x9k -n prod \
-o jsonpath='{.status.qosClass}'
# Guaranteed
How do you detect memory leaks?
A steadily climbing working set that always ends in OOMKilled is a leak, not an undersizing. Do not just raise the limit: find and fix the leak.
# Enable Native Memory Tracking to find non-heap leaks
java -XX:NativeMemoryTracking=summary -jar transfer-service.jar
# Inside the container, dump the NMT summary
jcmd 1 VM.native_memory summary
# Total: reserved=1500MB, committed=1100MB
# Java Heap: committed=768MB
# Thread: committed=150MB <- check thread count
# Internal: committed=80MB
# Direct: committed=200MB <- check for direct buffer leak
# If Direct grows over time without bound, it is a leak
# Common cause: Netty direct buffers not released
What are common mistakes?
The most common mistake is setting -Xmx equal to the container memory limit, which leaves zero room for off-heap memory.
# WRONG - heap equals container limit
# resources.limits.memory: 1Gi
# JAVA_OPTS: -Xmx1024m
# Result: OOMKilled under any load
# CORRECT - heap is 75% of container limit
# resources.limits.memory: 1Gi
# JAVA_OPTS: -XX:MaxRAMPercentage=75.0
# Result: 768 MB heap + 256 MB for off-heap
Another mistake is not graphing total container memory. If the dashboard only shows heap, you will never see the off-heap growth that causes OOM kills.
What should a senior engineer know?
CPU and memory behave differently. CPU is compressible: exceeding a CPU limit throttles the container. Memory is incompressible: exceeding a memory limit kills the container. That is why memory limits deserve the most care.
For a critical banking path, set requests equal to limits so the Pod gets Guaranteed QoS. Use Native Memory Tracking to profile non-heap memory when the heap graph looks innocent but OOM kills persist.
# Prometheus alert for OOMKilled containers
# Alert: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0
Visual Overview
JVM Memory Layout Inside a Container ====================================== Container Memory Limit: 1024 MB ┌─────────────────────────────────────┐ │ Java Heap (MaxRAMPercentage=75%) │ │ ┌─────────────────────────────┐ │ │ │ Young Gen + Old Gen │ │ │ │ ~768 MB max │ │ │ │ Dashboard shows THIS only │ │ │ └─────────────────────────────┘ │ │ │ │ Off-Heap / Native (~256 MB) │ │ ┌───────────┐ ┌───────────────┐ │ │ │ Thread │ │ Metaspace │ │ │ │ Stacks │ │ (classes) │ │ │ │ ~150 MB │ │ ~80 MB │ │ │ └───────────┘ └───────────────┘ │ │ ┌───────────┐ ┌───────────────┐ │ │ │ Direct │ │ Code Cache │ │ │ │ Buffers │ │ (JIT) │ │ │ │ ~variable │ │ ~60 MB │ │ │ └───────────┘ └───────────────┘ │ │ │ │ If Heap + Off-Heap > 1024 MB: │ │ → OOMKilled (exit 137) │ └─────────────────────────────────────┘
Key Takeaway
OOMKilled means the Linux kernel terminated the container because its total cgroup memory crossed the limit, and the reason it happens while monitoring looks normal is that most dashboards only graph the Java heap. The container limit caps the entire resident memory: heap plus thread stacks, metaspace, JIT code cache, direct byte buffers, and native allocations. So the heap can sit at a comfortable sixty percent while off-heap memory quietly pushes the total past the limit.
The JVM flags are central to avoiding this. Set MaxRAMPercentage to 75 percent so the heap uses three-quarters of the container limit and the remaining quarter is reserved for off-heap. Never set the heap equal to the whole limit, because that starves everything else and guarantees an OOM kill under load.
Right-sizing is a measurement exercise. The request should match the typical working set for scheduling, and the limit should sit above the observed peak of total container memory with headroom. For a critical banking service, setting request equal to limit gives Guaranteed QoS so the Pod is the last to be evicted under node pressure.
Remember one sentence: OOMKilled is about total memory, not just the heap, so set MaxRAMPercentage to 75 percent and graph container working set, not just heap.
---