What Exit Code 137 Means, and Telling OOMKill from Liveness-Probe Kill from Graceful-Shutdown Timeout | DevOps, CI/CD & Tooling

Q: What should a senior engineer know?

A tooshort grace period in a banking service can cut off inflight transfers midcommit. Size the grace period to the longest safe inflight operation and ensure the app listens for SIGTERM through a JVM shutdown hook. The preStop hook runs before SIGTERM and can be used to wait a few seconds for the l

Definition

137 is SIGKILL (128 plus 9) and only says force-killed; read describe and events to tell OOMKill from a liveness-probe kill from a grace-timeout kill, then fix memory, probe, or shutdown respectively.

Exit code 137 means the process was terminated by signal 9, SIGKILL.

Encoding – Linux reports signal termination as 128 + signal number; SIGKILL is 9, so 128 + 9 = 137.
SIGKILL – cannot be caught or handled; the process dies immediately with no cleanup.
Three causes in Kubernetes – OOMKill (breached memory limit), liveness-probe kill (failed probe threshold), graceful-shutdown timeout (did not exit within terminationGracePeriodSeconds).
Diagnosis – kubectl describe pod shows the Reason (OOMKilled vs other) and the Events timeline to distinguish the cause.

At the National Bank of India, exit code 137 is like a security guard physically carrying a teller out of the building. It only tells you the teller was removed by force, not why. Maybe they overflowed their cubicle with files (OOMKill), maybe the manager knocked and got no answer (liveness kill), or maybe closing time came and they dawdled past the deadline (shutdown timeout). The forced removal looks identical. You read the incident log to learn the real reason.

Understanding the Concepts

What does exit code 137 mean?

When a process is terminated by a signal, the shell and container runtime report the exit status as 128 plus the signal number. Signal 9 is SIGKILL, so 128 + 9 = 137. SIGKILL cannot be caught or handled, so the process dies immediately with no chance to clean up.

BASH

# Exit code 137 in Pod status
kubectl get pods -n prod
# NAME                              READY   STATUS      RESTARTS
# transfer-service-6d4f9c7b8a-x9k   0/1     Error       3

kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Last State:     Terminated
#   Exit Code:    137      <- 128 + 9 = SIGKILL
#   Reason:       ???      <- this tells you WHY

How do you identify an OOMKill?

When a container exceeds its cgroup memory limit, the kernel sends SIGKILL. The describe output shows Reason: OOMKilled.

BASH

kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Last State:     Terminated
#   Reason:       OOMKilled     <- THIS is the indicator
#   Exit Code:    137
#   Started:      2025-01-15T10:00:00Z
#   Finished:     2025-01-15T10:45:00Z

# Fix: right-size memory and make the JVM container-aware

YAML

# Fix for OOMKill: container-aware JVM + right-sized limits
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transfer-service
spec:
  template:
    spec:
      containers:
        - name: transfer
          resources:
            requests:
              memory: 768Mi
            limits:
              memory: 1Gi
          env:
            - name: JAVA_OPTS
              value: "-XX:MaxRAMPercentage=75.0"

How do you identify a liveness-probe kill?

When a liveness probe fails past its failureThreshold, the kubelet kills and restarts the container. The events show Unhealthy liveness events before the Killing event.

BASH

kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Last State:     Terminated
#   Reason:       Error          <- NOT OOMKilled
#   Exit Code:    137
# Events:
#   Warning  Unhealthy  Liveness probe failed: HTTP probe failed (503)
#   Warning  Unhealthy  Liveness probe failed: HTTP probe failed (503)
#   Warning  Unhealthy  Liveness probe failed: HTTP probe failed (503)
#   Normal   Killing    Container transfer failed liveness probe, will be restarted

# The restart count climbs with probe failures, not with memory
kubectl get pods -n prod
# RESTARTS: 5   <- correlates with liveness failures

YAML

# Fix: make liveness a trivial self-check, not a downstream check
# WRONG - liveness depends on the database
livenessProbe:
  httpGet:
    path: /actuator/health   # includes DB, Redis
    port: 8080
  failureThreshold: 2

# CORRECT - liveness is a self-check, readiness checks downstream
livenessProbe:
  httpGet:
    path: /actuator/health/liveness   # trivial self-check
    port: 8080
  failureThreshold: 3
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /actuator/health/readiness  # includes DB, Redis
    port: 8080
  failureThreshold: 3
  periodSeconds: 5

How do you identify a graceful-shutdown-timeout kill?

When a Pod is deleted or rescheduled, the kubelet sends SIGTERM and waits up to terminationGracePeriodSeconds. If the process does not exit in time, the kubelet sends SIGKILL. This only happens during deploys, scale-downs, or evictions.

BASH

kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Events:
#   Normal   Killing    Stopping container transfer
#   Normal   Killing    Container transfer did not terminate within grace period,
#                       sending SIGKILL
# Last State:     Terminated
#   Reason:       Error
#   Exit Code:    137

# This happens during deploys, not randomly
# The grace period was 30s but the app needed 45s to drain

YAML

# Fix: handle SIGTERM + size the grace period to the longest in-flight operation
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transfer-service
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60   # give 60s to drain
      containers:
        - name: transfer
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]  # let LB stop sending traffic

JAVA

// Handle SIGTERM with a JVM shutdown hook
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    log.info("SIGTERM received, draining in-flight transfers...");
    transferService.stopAcceptingNew();
    transferService.drainInFlight(Duration.ofSeconds(45));
    log.info("Drain complete, exiting cleanly.");
}));

How do these three causes compare?

| Cause | Reason in describe | Events before kill | When it happens | Fix |
|---|---|---|---|---|
| OOMKill | OOMKilled | None (kernel kill) | Under load, memory spike | Right-size limits, MaxRAMPercentage |
| Liveness-probe kill | Error | Unhealthy liveness events | Random, correlates with probe failures | Trivial liveness endpoint |
| Shutdown timeout | Error | SIGTERM then SIGKILL | During deploys/evictions | Handle SIGTERM, size grace period |

What about exit code 143?

Exit code 143 means SIGTERM (128 + 15). This is a graceful termination where the process had a chance to clean up. A healthy shutdown during a deploy produces 143, not 137.

BASH

# Healthy shutdown: SIGTERM received, process exits cleanly
# Exit Code: 143  <- 128 + 15 = SIGTERM
# This is expected during deploys and scale-downs

# Unhealthy shutdown: SIGTERM received, process did not exit in time
# Exit Code: 137  <- 128 + 9 = SIGKILL (forced after grace period)

What are common mistakes?

The most common mistake is treating every 137 the same way. Each cause requires a different fix.

BASH

# WRONG - see 137, raise the memory limit
# But the cause was a liveness probe kill, not OOM
# Raising memory does nothing

# WRONG - see 137, loosen the liveness probe
# But the cause was OOM, not a probe kill
# Loosening the probe does nothing

# CORRECT - read describe first, identify the cause, apply the right fix
kubectl describe pod transfer-service-x9k -n prod
# Read the Reason and Events, then apply the matching fix

What should a senior engineer know?

A too-short grace period in a banking service can cut off in-flight transfers mid-commit. Size the grace period to the longest safe in-flight operation and ensure the app listens for SIGTERM through a JVM shutdown hook.

The preStop hook runs before SIGTERM and can be used to wait a few seconds for the load balancer to stop sending traffic, preventing requests that arrive after SIGTERM but before the LB updates.

YAML

# Production-grade shutdown configuration
spec:
  terminationGracePeriodSeconds: 90
  containers:
    - name: transfer
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]   # wait for LB drain

BASH

# Monitor for 137 exits with Prometheus
# Alert: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0
# Alert: increase(kube_pod_container_status_restarts_total[5m]) > 3

Visual Overview

DIAGRAM

  Exit Code 137 Decision Tree
  =============================

  kubectl describe pod → Last State
       │
       ├── Reason: OOMKilled
       │     └─→ Container breached memory limit
       │         Fix: right-size limits + MaxRAMPercentage=75
       │              or fix memory leak (NMT)
       │
       ├── Reason: Error + Events show liveness probe failures
       │     └─→ Liveness probe failed past threshold
       │         Fix: trivial liveness endpoint
       │              separate liveness from readiness
       │
       └── Reason: Error + Events show SIGTERM then SIGKILL
             └─→ Process did not exit within grace period
                 Fix: handle SIGTERM (shutdown hook)
                      size terminationGracePeriodSeconds
                      add preStop hook for LB drain


  Exit Code Cheat Sheet
  ======================
  137 = 128 + 9  = SIGKILL (forced, no cleanup)
  143 = 128 + 15 = SIGTERM (graceful, cleanup possible)
  1   = Application error (exception, bad config)
  0   = Normal exit (Job completed successfully)

Key Takeaway

Exit code 137 decodes as 128 plus signal 9, SIGKILL, so the number tells you only that the process was force-killed. The crucial point is that 137 is a symptom shared by three different causes, so never act on the number alone. Read the describe output and the events timeline to identify the actual reason.

An OOMKill shows Reason: OOMKilled, meaning the container breached its cgroup memory limit. Fix it with right-sized memory limits and container-aware JVM flags. A liveness-probe kill shows Unhealthy liveness events before a Killing event. Fix it by making the liveness endpoint a trivial self-check that does not depend on slow downstreams. A graceful-shutdown-timeout kill shows SIGTERM followed by SIGKILL after the grace period. Fix it by handling SIGTERM in a shutdown hook and sizing the grace period to the longest in-flight operation.

In a banking service, the shutdown timeout is the most dangerous of the three because it can cut off in-flight transfers mid-commit. Size the grace period generously and always implement a proper SIGTERM handler.

Remember one sentence: 137 means SIGKILL, but read the Reason and Events to tell OOMKill from liveness-probe kill from shutdown timeout, because each has a completely different fix.

---