Kubernetes QoS Classes, OOM Eviction Order, and Why emptyDir Data Is Lost | DevOps, CI/CD & Tooling

Q: What about emptyDir with medium: Memory?

An emptyDir with medium: Memory uses tmpfs (RAMbacked filesystem). It is extremely fast but counts against the container's memory limit, and its contents vanish on Pod removal just like a diskbacked emptyDir. RAMbacked emptyDir for highspeed scratch volumes: name: fastcache emptyDir: medium: Memory

Definition

Guaranteed Pods survive longest and BestEffort die first under memory pressure, and emptyDir is scratch space tied to the Pod, so it vanishes on restart or reschedule.

Kubernetes assigns each Pod a Quality of Service class based on resource requests and limits.

Guaranteed – every container sets requests equal to limits for both CPU and memory; protected from eviction.
Burstable – at least one container sets a request, but not all equal limits; evicted after BestEffort.
BestEffort – no requests or limits at all; evicted first under node pressure.
Eviction order – BestEffort first, then Burstable exceeding requests, then Guaranteed last.
emptyDir – a scratch volume tied to the Pod lifecycle; created on scheduling, deleted on Pod removal or reschedule.

At the National Bank of India, when the branch runs short of space, management decides who gets moved out first. BestEffort tellers made no reservation, so they go first. Burstable tellers reserved a small desk but overflowed, so they go next. Guaranteed tellers reserved exactly the space they use, so they are protected. An emptyDir is a scratch notepad clipped to the teller's badge: the moment that teller leaves the building the notepad is thrown away.

Understanding the Concepts

How are QoS classes determined?

The QoS class is derived, not set directly. Kubernetes inspects the requests and limits on every container in the Pod and assigns the class automatically.

YAML

# Guaranteed: requests == limits for all containers, both CPU and memory
apiVersion: v1
kind: Pod
metadata:
  name: transfer-service
spec:
  containers:
    - name: transfer
      resources:
        requests:
          cpu: "1"
          memory: 1Gi
        limits:
          cpu: "1"
          memory: 1Gi

YAML

# Burstable: at least one request or limit, but not all equal
apiVersion: v1
kind: Pod
metadata:
  name: report-generator
spec:
  containers:
    - name: report
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
        limits:
          cpu: "1"
          memory: 1Gi

YAML

# BestEffort: no requests or limits at all
apiVersion: v1
kind: Pod
metadata:
  name: debug-pod
spec:
  containers:
    - name: debug
      image: curlimages/curl
      # no resources section

BASH

# Check the QoS class
kubectl get pod transfer-service -o jsonpath='{.status.qosClass}'
# Guaranteed

kubectl get pod report-generator -o jsonpath='{.status.qosClass}'
# Burstable

kubectl get pod debug-pod -o jsonpath='{.status.qosClass}'
# BestEffort

How does QoS affect eviction order?

Under node memory pressure, the kubelet evicts Pods in this order: BestEffort first (reserved nothing), then Burstable that are using more than their requests, and Guaranteed last. The Linux OOM killer assigns each process an oom_score_adj that Kubernetes sets according to QoS.

| QoS Class | oom_score_adj | Eviction priority | Behaviour under pressure |
|---|---|---|---|
| BestEffort | 1000 | First to die | No reservation, no protection |
| Burstable | 2-999 (proportional) | Second to die | Evicted if using above request |
| Guaranteed | -997 | Last to die | Fully reserved, maximum protection |

BASH

# Scenario: node runs out of memory
# The kubelet evicts in this order:
# 1. debug-pod (BestEffort) - killed first
# 2. report-generator (Burstable, using 900Mi with 512Mi request) - killed next
# 3. transfer-service (Guaranteed) - protected, killed only as last resort

# Verify oom_score_adj inside a container
kubectl exec transfer-service -- cat /proc/1/oom_score_adj
# -997  <- Guaranteed, maximum protection

How do CPU and memory differ in enforcement?

CPU is compressible: exceeding a CPU limit throttles the container. Memory is incompressible: exceeding a memory limit kills the container. This is why memory limits deserve more care than CPU limits.

BASH

# CPU throttling: container slows down but keeps running
# CFS quota throttling visible in metrics:
# container_cpu_cfs_throttled_seconds_total > 0

# Memory limit breach: container is OOMKilled
# Exit code: 137 (128 + SIGKILL)
kubectl describe pod transfer-service
# Last State: Terminated, Reason: OOMKilled, Exit Code: 137

Why is emptyDir data lost?

An emptyDir volume is created empty when the Pod is scheduled to a node. Its lifetime is exactly the Pod's lifetime on that node.

A container restart within the Pod keeps the emptyDir, because the Pod is still the same Pod on the same node.
A Pod deletion, eviction, or reschedule destroys the emptyDir and all its contents, because a brand-new Pod gets a brand-new empty volume.

YAML

# emptyDir for scratch/cache data - data lost on Pod reschedule
apiVersion: v1
kind: Pod
metadata:
  name: statement-generator
spec:
  containers:
    - name: generator
      volumeMounts:
        - name: cache
          mountPath: /tmp/cache
  volumes:
    - name: cache
      emptyDir: {}   # created fresh on every Pod scheduling

BASH

# Container restarts: emptyDir survives
kubectl exec statement-generator -- ls /tmp/cache
# cached-statement-1.pdf  cached-statement-2.pdf

# Pod is rescheduled to another node:
kubectl delete pod statement-generator
# New Pod starts on a different node
kubectl exec statement-generator -- ls /tmp/cache
# (empty - all cached files are gone)

When should you use emptyDir vs PersistentVolumeClaim?

emptyDir is correct for scratch data. PersistentVolumeClaim is correct for anything that must persist.

| Aspect | emptyDir | PersistentVolumeClaim |
|---|---|---|
| Lifecycle | Tied to Pod | Independent of Pod |
| Survives container restart? | Yes | Yes |
| Survives Pod reschedule? | No | Yes |
| Survives node loss? | No | Yes (if backed by EBS/EFS) |
| Use case | Cache, temp downloads | Ledger data, uploaded documents |

YAML

# PersistentVolumeClaim for durable banking data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ledger-data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3
  resources:
    requests:
      storage: 100Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: ledger-db
spec:
  containers:
    - name: postgres
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: ledger-data   # survives Pod restarts and reschedules

What about emptyDir with medium: Memory?

An emptyDir with medium: Memory uses tmpfs (RAM-backed filesystem). It is extremely fast but counts against the container's memory limit, and its contents vanish on Pod removal just like a disk-backed emptyDir.

YAML

# RAM-backed emptyDir for high-speed scratch
volumes:
  - name: fast-cache
    emptyDir:
      medium: Memory
      sizeLimit: 256Mi   # counts against container memory limit

What are common mistakes?

The most common mistake is running a critical banking service as BestEffort. Without requests or limits, it is the first to be evicted when a noisy batch job causes node pressure.

YAML

# WRONG - no resources, BestEffort QoS
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transfer-service
spec:
  template:
    spec:
      containers:
        - name: transfer
          image: nbi-registry.example.com/transfer-service:2.0.0
          # no resources section -> BestEffort -> evicted first

# CORRECT - Guaranteed QoS for the critical path
          resources:
            requests:
              cpu: "1"
              memory: 1Gi
            limits:
              cpu: "1"
              memory: 1Gi

Another mistake is storing durable data in an emptyDir, thinking it persists. Authoritative data like bank statements or ledger records must live in a PersistentVolume.

What should a senior engineer know?

The senior discipline is to match the QoS class to the service's importance and the volume type to the durability requirement. Critical banking services get Guaranteed QoS. Batch jobs and internal tools can be Burstable. Debug Pods can be BestEffort.

For StatefulSets, the volumeClaimTemplates create a per-replica PVC that survives Pod restarts and reschedules, which is why StatefulSets are used for databases.

BASH

# List QoS classes for all Pods in the banking namespace
kubectl get pods -n banking \
  -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
# NAME                     QOS
# transfer-service-x9k     Guaranteed
# report-generator-abc     Burstable
# debug-pod                BestEffort

Visual Overview

DIAGRAM

  QoS Eviction Order Under Node Memory Pressure
  ===============================================

  Node memory pressure detected
       │
       ▼
  ┌─────────────────────────────────────────┐
  │  Step 1: Evict BestEffort Pods           │
  │  (oom_score_adj = 1000)                  │
  │  No reservation, first to go             │
  │  [debug-pod] → EVICTED                   │
  └────────────┬────────────────────────────┘
               │ still under pressure?
               ▼
  ┌─────────────────────────────────────────┐
  │  Step 2: Evict Burstable Pods            │
  │  (oom_score_adj = 2-999)                 │
  │  Only if using above their request       │
  │  [report-generator: 900Mi/512Mi] → EVICT │
  └────────────┬────────────────────────────┘
               │ still under pressure?
               ▼
  ┌─────────────────────────────────────────┐
  │  Step 3: Evict Guaranteed Pods (last)    │
  │  (oom_score_adj = -997)                  │
  │  Fully reserved, maximum protection      │
  │  [transfer-service] → protected          │
  └─────────────────────────────────────────┘


  emptyDir Lifecycle
  ===================

  Pod scheduled to Node-1
       │
       ▼  emptyDir created (empty)
  Pod runs, writes cache files
       │
       │  Container crashes and restarts
       │  emptyDir SURVIVES (same Pod)
       │
       │  Pod deleted / rescheduled
       ▼  emptyDir DESTROYED (gone)
  New Pod on Node-2 gets a new empty emptyDir

Key Takeaway

Kubernetes assigns each Pod a Quality of Service class based on resource requests and limits. Guaranteed means requests equal limits for all containers, and these Pods are the last to be evicted under node memory pressure. Burstable means some requests are set but not all equal limits. BestEffort means no requests or limits at all, and these Pods die first. The practical lesson is to make critical banking services Guaranteed so a noisy batch job cannot cause the kubelet to evict the payment service.

CPU and memory differ in enforcement. CPU is compressible, so exceeding a limit only throttles the container. Memory is incompressible, so exceeding a limit kills the container outright. That is why memory requests and limits deserve the most care.

An emptyDir volume shares the Pod's lifecycle. A container restart keeps the emptyDir because the Pod is still the same Pod, but if the Pod is deleted, evicted, or rescheduled, the emptyDir and everything in it are gone. For durable banking data use a PersistentVolumeClaim, whose lifecycle is independent of any Pod.

Remember one sentence: make critical services Guaranteed so they survive eviction, and use PersistentVolumeClaims for data that must outlive a Pod.

---