(Scenario) Diagnosing CrashLoopBackOff, ImagePullBackOff, Pending, and ContainerCreating | DevOps, CI/CD & Tooling

Definition

CrashLoopBackOff is the app crashing after start, ImagePullBackOff is the image not pulling, Pending is the scheduler finding no room, and ContainerCreating is setup in progress, each diagnosed from describe events and logs.

These are four distinct Pod phases that signal four different failure classes.

CrashLoopBackOff – the container starts and then exits repeatedly with exponential backoff; the application itself is failing.
ImagePullBackOff – the kubelet cannot pull the container image; usually a wrong tag, missing image, or missing registry credentials.
Pending – the scheduler cannot place the Pod on any node; usually insufficient CPU/memory, taints, or an unbound PersistentVolumeClaim.
ContainerCreating – the Pod is scheduled but the kubelet is still setting it up; typically waiting on image pull, volume mount, or a missing Secret.
Diagnostic tools – kubectl describe pod for events, kubectl logs --previous for the last crash output.

At the National Bank of India, CrashLoopBackOff is a teller who sits down and immediately faints, over and over. ImagePullBackOff is a teller who cannot even get their uniform because the supply room door is locked. Pending is a teller with no desk because the branch is full. ContainerCreating is a teller who has a desk and is still unpacking their drawer. Each needs a completely different fix, so the first job is to read the incident log to learn which story you are in.

Understanding the Concepts

How do you diagnose CrashLoopBackOff?

The container launches, the process exits, the kubelet restarts it with exponential backoff, and the state cycles. The key tool is kubectl logs --previous because the current container may be too young to have logs.

BASH

# Check Pod status
kubectl get pods -n prod
# NAME                              READY   STATUS             RESTARTS
# transfer-service-6d4f9c7b8a-x9k   0/1     CrashLoopBackOff   5

# View the crashed container's logs
kubectl logs transfer-service-6d4f9c7b8a-x9k --previous -n prod
# org.postgresql.util.PSQLException: Connection to ledger-db:5432 refused
# at com.bank.service.TransferService.init(TransferService.java:45)
# Application failed to start

# Read events and last state
kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Last State:  Terminated
#   Reason:    Error
#   Exit Code: 1
# Events:
#   Warning  BackOff  Back-off restarting failed container

Common banking causes: a wrong JDBC URL, a missing Secret (DB password not mounted), a failed Flyway migration at boot, or an out-of-memory kill immediately after startup.

How do you diagnose ImagePullBackOff?

The scheduler placed the Pod but the kubelet cannot fetch the image. The describe events show the exact reason.

BASH

kubectl get pods -n prod
# NAME                              READY   STATUS             RESTARTS
# transfer-service-6d4f9c7b8a-x9k   0/1     ImagePullBackOff   0

kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Events:
#   Warning  Failed   Failed to pull image "nbi-registry.example.com/transfer-service:2.1.0":
#            rpc error: manifest for transfer-service:2.1.0 not found
#   Warning  Failed   Error: ImagePullBackOff

Common causes and fixes:

BASH

# Cause 1: Typo in image name or tag
# Fix: correct the image reference
kubectl set image deployment/transfer-service \
  transfer=nbi-registry.example.com/transfer-service:2.1.0

# Cause 2: Image tag was never pushed to the registry
# Fix: push the image first
docker push nbi-registry.example.com/transfer-service:2.1.0

# Cause 3: Missing registry credentials (private bank registry)
# Fix: create and attach an imagePullSecret
kubectl create secret docker-registry nbi-registry-creds \
  --docker-server=nbi-registry.example.com \
  --docker-username=deploy-bot \
  --docker-password=secret-token \
  -n prod

YAML

# Attach the pull secret to the Pod spec
spec:
  imagePullSecrets:
    - name: nbi-registry-creds
  containers:
    - name: transfer
      image: nbi-registry.example.com/transfer-service:2.1.0

How do you diagnose Pending?

The Pod is not scheduled at all. The describe events explain exactly why.

BASH

kubectl get pods -n prod
# NAME                              READY   STATUS    RESTARTS
# transfer-service-6d4f9c7b8a-new   0/1     Pending   0

kubectl describe pod transfer-service-6d4f9c7b8a-new -n prod
# Events:
#   Warning  FailedScheduling  0/5 nodes are available:
#     2 Insufficient cpu, 3 Insufficient memory

# Compare Pod requests to node capacity
kubectl describe node ip-10-0-1-10 | grep -A5 Allocated
# Allocated resources:
#   CPU Requests:   3800m (95%)
#   Memory Requests: 7Gi (90%)

Common causes and fixes:

BASH

# Cause 1: Resource requests too high for available nodes
# Fix: lower requests or let Cluster Autoscaler add nodes
kubectl get nodes
# Wait for Cluster Autoscaler to add a node

# Cause 2: Node selector or affinity matches nothing
kubectl get nodes --show-labels | grep zone
# Fix: correct the node selector to match existing labels

# Cause 3: Taint the Pod does not tolerate
kubectl describe node ip-10-0-1-10 | grep Taints
# Fix: add a toleration to the Pod spec

# Cause 4: Unbound PersistentVolumeClaim
kubectl get pvc -n prod
# NAME      STATUS    VOLUME   CAPACITY
# data-db   Pending   <none>   <none>
# Fix: provision a matching PersistentVolume or StorageClass

How do you diagnose ContainerCreating?

The Pod is scheduled and the kubelet is working through setup. The describe events reveal what it is waiting on.

BASH

kubectl get pods -n prod
# NAME                              READY   STATUS              RESTARTS
# transfer-service-6d4f9c7b8a-x9k   0/1     ContainerCreating   0

kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Events:
#   Normal   Scheduled  Successfully assigned prod/transfer-service to ip-10-0-1-10
#   Warning  Failed     MountVolume.SetUp failed: secret "db-credentials" not found

Common causes:

BASH

# Cause 1: Referenced Secret or ConfigMap does not exist
kubectl get secrets -n prod | grep db-credentials
# (empty - the Secret is missing)
# Fix: create the Secret
kubectl create secret generic db-credentials \
  --from-literal=DB_PASSWORD=securePass -n prod

# Cause 2: Volume will not attach (cloud disk in wrong AZ)
# Events: AttachVolume.Attach failed: volume is in us-east-1a, node is in us-east-1b
# Fix: ensure the PV and node are in the same availability zone

# Cause 3: Slow image pull (large image, slow registry)
# Events: Pulling image "nbi-registry.example.com/transfer-service:2.1.0"
# Fix: use smaller multi-stage images, or a registry closer to the cluster

How do these four states compare?

| State | Pod scheduled? | Container running? | Root cause layer |
|---|---|---|---|
| CrashLoopBackOff | Yes | Starts then exits | Application code / config |
| ImagePullBackOff | Yes | Never starts | Image registry / credentials |
| Pending | No | N/A | Scheduler / resources / PVC |
| ContainerCreating | Yes | Setting up | Kubelet / volumes / secrets |

What is the systematic debugging approach?

The discipline across all four states is identical: always start with kubectl describe pod to read the Events section. The events almost always name the exact cause. Only then reach for logs, node inspection, or volume checks.

BASH

# Step 1: Always start here
kubectl describe pod <pod-name> -n prod
# Read the Events section carefully

# Step 2: For CrashLoopBackOff - read previous container logs
kubectl logs <pod-name> --previous -n prod

# Step 3: For Pending - check node capacity
kubectl describe nodes | grep -A10 "Allocated resources"

# Step 4: For ImagePullBackOff - verify image and credentials
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].image}'
kubectl get pod <pod-name> -o jsonpath='{.spec.imagePullSecrets}'

# Step 5: For ContainerCreating - check volumes and secrets
kubectl get pvc -n prod
kubectl get secrets -n prod

What are common mistakes?

The most common mistake is restarting Pods blindly hoping the problem goes away, instead of reading the events to understand the actual cause.

BASH

# WRONG - restart without understanding
kubectl delete pod transfer-service-6d4f9c7b8a-x9k -n prod
# The new Pod hits the same problem

# CORRECT - read events first, fix the root cause
kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Events reveal: Secret "db-credentials" not found
kubectl create secret generic db-credentials \
  --from-literal=DB_PASSWORD=securePass -n prod
# Now the Pod progresses past ContainerCreating

What should a senior engineer know?

A senior engineer knows that CrashLoopBackOff with exit code 137 means OOMKilled, not just a crash. Check the Last State reason in describe output.

For ImagePullBackOff in a private bank registry, the imagePullSecrets must be attached to the Pod's service account, not just to the Pod spec, so that every Pod in the namespace automatically uses the credentials.

BASH

# Attach pull secret to the default service account
kubectl patch serviceaccount default -n prod \
  -p '{"imagePullSecrets": [{"name": "nbi-registry-creds"}]}'

For Pending Pods, the Cluster Autoscaler only helps if the Pod's requests fit on a node that the autoscaler can provision. If the request exceeds the largest instance type's allocatable capacity, no amount of autoscaling will help.

Visual Overview

DIAGRAM

  Pod State Diagnostic Decision Tree
  ====================================

  kubectl get pods → STATUS column
       │
       ├── CrashLoopBackOff
       │     └─→ kubectl logs --previous
       │         └─→ Read stack trace / error message
       │             └─→ Fix: config, code, DB connection, memory
       │
       ├── ImagePullBackOff
       │     └─→ kubectl describe pod → Events
       │         └─→ "manifest not found" or "unauthorized"
       │             └─→ Fix: correct tag, push image, add pull secret
       │
       ├── Pending
       │     └─→ kubectl describe pod → FailedScheduling
       │         └─→ "Insufficient cpu/memory" or "no matching node"
       │             └─→ Fix: lower requests, add nodes, fix selector/taint
       │
       └── ContainerCreating
             └─→ kubectl describe pod → Events
                 └─→ "Secret not found" or "volume attach failed"
                     └─→ Fix: create Secret, provision volume, check AZ

Key Takeaway

These four Pod states each point to a different layer of failure, and the diagnostic approach is always the same: run kubectl describe pod and read the Events section first. The events almost always name the exact cause before you touch anything else.

CrashLoopBackOff means the container launches and then the process exits repeatedly, so the previous container's logs hold the stack trace or error message. ImagePullBackOff means the kubelet cannot fetch the image, and the describe event tells you whether it is a wrong tag, a missing image, or missing registry credentials. Pending means the scheduler could not place the Pod, and the FailedScheduling event explains whether it is insufficient resources, a taint, or an unbound PVC. ContainerCreating means the kubelet is still setting up the Pod, usually waiting on a missing Secret, a volume that will not attach, or a slow image pull.

The common thread is discipline: read the events first, then reach for logs, node capacity, or volume checks depending on which state the Pod is stuck in. Never restart Pods blindly hoping the problem goes away.

Remember one sentence: always start with kubectl describe pod and read the Events, because the events name the cause and point to the fix.

---