Back to Docker: Images, Containers, Multi-Stage
Q7Question 7 of 10

Scenario: A Container Keeps Restarting in Production: Finding the Root Cause

Key Concepts
  • A restarting container means the process is dying or a health probe is failing; in Kubernetes this shows as CrashLoopBackOff.
  • Read the previous container's logs first with kubectl logs --previous, because the current instance may have no output yet.
  • Exit code 137 with reason OOMKilled means the process exceeded its memory limit.
  • A failing liveness probe causes restarts even when the app is healthy; check the probe path, port, and timing.
  • Exit code 1 or 2 at startup usually means a missing config, secret, or environment variable.
  • Capture the failed pod's logs and describe output before it is replaced, because the evidence disappears.

Definition

A restarting container is a dying or probe-failing process, so read its previous logs and exit code, check for OOMKilled, check probe configuration, and verify the entrypoint and config.
  • CrashLoopBackOff – Kubernetes state where a container keeps crashing and the system restarts it with increasing backoff delays.
  • Exit codes – 0 is clean exit (bug for a long-running service), 137 is OOMKilled (SIGKILL), 139 is segfault, 143 is SIGTERM.
  • OOMKilled – the kernel terminated the process for exceeding its memory limit (cgroup).
  • Liveness probe – a failing liveness probe makes the kubelet kill and restart the container.
  • Previous logs – kubectl logs --previous shows the crashed instance's output before the restart.

Imagine a new teller at the National Bank of India who keeps walking out of the branch minutes after arriving, and the manager keeps hiring a replacement. To find out why, check the exit note (logs and exit code): did they collapse from overwork (OOMKilled), did the inspector keep declaring them unfit (failing liveness probe), or did they show up to the wrong address (bad entrypoint or missing config)?

Understanding the Concepts

What does a restarting container look like?

In Kubernetes, a container that keeps crashing shows as CrashLoopBackOff with increasing restart delays.

BASH
kubectl get pods -n prod -l app=transfer-service
# NAME                                READY   STATUS             RESTARTS   AGE
# transfer-service-7d4c8-abc12        0/1     CrashLoopBackOff   5          3m

In plain Docker, you see the container repeatedly in Restarting state.

BASH
docker ps -a | grep transfer
# CONTAINER ID   STATUS                          NAMES
# abc123         Restarting (137) 10 seconds ago  transfer-service

How do you read the logs?

Start with the previous container's logs, because the current instance may have crashed before producing output.

BASH
# Kubernetes: logs from the crashed instance
kubectl logs transfer-service-7d4c8-abc12 -n prod --previous
# java.lang.OutOfMemoryError: Java heap space
#   at com.bank.service.TransferService.processLargeBatch

# Docker: logs from the container
docker logs transfer-service
# Or docker logs --tail 100 transfer-service

The logs usually name the cause directly: a stack trace, a connection refused, a missing property.

How do you read the exit code?

The exit code classifies how the process ended.

BASH
# Kubernetes: describe shows the last state and exit code
kubectl describe pod transfer-service-7d4c8-abc12 -n prod
# Last State:     Terminated
#   Reason:       OOMKilled
#   Exit Code:    137

# Docker: inspect shows the exit code
docker inspect transfer-service --format '{{.State.ExitCode}}'
# 137

| Exit Code | Signal | Meaning | Common Cause |
|---|---|---|---|
| 0 | Normal | Clean exit | Bug: long-running service should not exit |
| 1 | Error | Application error | Missing config, unhandled exception |
| 137 | SIGKILL | Killed by kernel or orchestrator | OOMKilled or liveness probe |
| 139 | SIGSEGV | Segmentation fault | Native library crash |
| 143 | SIGTERM | Graceful termination | Kubelet stopping the pod |

What is OOMKilled and how do you fix it?

OOMKilled means the process exceeded its cgroup memory limit. The kernel's OOM killer terminates it.

BASH
# Confirm OOMKilled
kubectl describe pod transfer-service-pod -n prod
# Reason: OOMKilled
# Exit Code: 137

# Check current memory usage vs limit
kubectl top pod transfer-service-pod -n prod
# NAME                    CPU    MEMORY
# transfer-service-pod    50m    490Mi   (limit: 512Mi)
YAML
# Fix 1: raise the memory limit if the app genuinely needs more.
resources:
  limits:
    memory: "1Gi"   # Was 512Mi

# Fix 2: for Java, align the JVM heap to the container limit.
# WRONG: JVM heap larger than container limit.
ENTRYPOINT ["java", "-Xmx768m", "-jar", "app.jar"]
# Container limit: 512Mi, JVM tries to use 768 MB → OOMKilled.

# CORRECT: set max heap as a percentage of the container limit.
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75.0", "-jar", "app.jar"]
# JVM uses at most 75% of the 512Mi limit = ~384 MB heap,
# leaving room for non-heap memory (metaspace, threads, native).

What if a liveness probe is causing the restarts?

A liveness probe that fails makes the kubelet kill and restart the container, even if the app is healthy. Common causes: wrong path, wrong port, or the probe fires before the app starts.

YAML
# WRONG: liveness probe fires during startup, killing a healthy app.
livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  initialDelaySeconds: 5   # App takes 30 seconds to start!
  periodSeconds: 10
  failureThreshold: 3
# The probe fails 3 times during startup → container killed → CrashLoopBackOff.

# CORRECT: use a startup probe for slow-starting apps.
startupProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  periodSeconds: 5
  failureThreshold: 30   # Allows up to 150 seconds for startup.
livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
# Startup probe protects the app during startup.
# Liveness probe takes over only after startup succeeds.

What if the entrypoint or config is wrong?

A container that can never start exits immediately with a non-zero code.

BASH
# Previous logs show the error
kubectl logs transfer-service-pod -n prod --previous
# Error: Required key 'DB_PASSWORD' not found in environment
# Or: java: command not found (bad entrypoint)
# Or: Error: Unable to access jarfile /app/wrong-name.jar
BASH
# Fix: create the missing secret
kubectl create secret generic transfer-secrets \
  --from-literal=DB_PASSWORD='s3cur3p@ss' -n prod

# Fix: correct the entrypoint in the Dockerfile
ENTRYPOINT ["java", "-jar", "/app/transfer-service.jar"]

How do you separate liveness from readiness?

A common mistake is confusing the two. A readiness failure should remove the pod from the load balancer but NOT restart it. A liveness failure restarts.

YAML
# readinessProbe: controls traffic. Failure = no traffic, no restart.
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080

# livenessProbe: controls restarts. Failure = kill and restart.
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
TEXT
# WRONG: using liveness for what should be readiness.
# A temporary database outage causes the liveness probe to fail,
# which restarts the container. But restarting does not fix the DB.

# CORRECT: readiness fails on DB outage (pod removed from LB).
# Liveness only fails if the process is truly stuck (deadlock).

How do you preserve evidence?

Once the pod is recycled, the logs and events disappear.

BASH
# Capture evidence before rollback
kubectl logs transfer-service-pod -n prod --previous > /tmp/crashed.log
kubectl describe pod transfer-service-pod -n prod > /tmp/crashed-describe.txt

# Then investigate and fix

What should a senior engineer know?

  • Modern JVMs are container-aware: use -XX:MaxRAMPercentage=75.0 instead of a fixed -Xmx to align heap to the cgroup limit.
  • A startup probe protects slow-starting apps. Without it, the liveness probe can kill the app before it finishes starting.
  • Resist the urge to just bump memory or restart without understanding the exit code. That masks leaks that will recur.
  • CrashLoopBackOff is a symptom, not a cause. The backoff is Kubernetes spacing out restarts.
BASH
# Debug order summary
# 1. kubectl logs --previous (what did the app say?)
# 2. kubectl describe pod (exit code, reason, events)
# 3. kubectl top pod (memory near limit?)
# 4. Check probe config (wrong path, port, timing?)
# 5. Check secrets and config (missing?)

Visual Overview

DIAGRAM
  Container keeps restarting:

  ┌─────────────────────────────────┐
  │  1. kubectl logs --previous      │
  │  What did the app say?           │
  └──────────────┬──────────────────┘
                 ▼
  ┌─────────────────────────────────┐
  │  2. kubectl describe pod         │
  │  Exit code? OOMKilled? Events?   │
  └──────────────┬──────────────────┘
                 ▼
  ┌────────┬─────────────┬──────────┐
  │ 137    │ 1 or 2      │ 143      │
  │OOMKill │ App error   │ Probe    │
  │        │ (missing    │ killing  │
  │        │  config)    │ healthy  │
  │        │             │ app      │
  └───┬────┴──────┬──────┴────┬─────┘
      ▼           ▼           ▼
  Raise limit  Fix config   Fix probe
  or fix heap  or secret    path/timing

Key Takeaway

A container that keeps restarting is a process that is dying or a health probe that keeps failing. In Kubernetes it shows as CrashLoopBackOff with increasing delays between restarts. I follow a fixed debug order: first read the previous instance's logs, which usually contain the real error; then inspect the exit code and reason from kubectl describe to classify the failure.

The three most common causes each have a specific fix. OOMKilled (exit 137) means the process exceeded its memory limit, fixed by raising the limit or aligning the JVM heap to the cgroup with MaxRAMPercentage. A failing liveness probe (exit 137/143) kills a healthy app because the probe path, port, or timing is wrong, fixed by correcting the probe or adding a startup probe for slow-starting apps. A non-zero startup exit (exit 1) usually means a missing secret or config, named directly in the previous logs.

The senior points are to separate liveness from readiness so a temporary database outage does not trigger needless restarts, to capture the failed pod's evidence before it is recycled, and to resist blindly bumping memory or restarting without understanding the exit code.

Remember one sentence: read the previous logs, classify the exit code, and fix the specific cause rather than blindly restarting.

---

Share
Use ← → arrow keys to navigate