Back to CI/CD Pipeline
Q41Question 41 of 56

Scenario: CI Succeeds but CD Fails: Debugging, Where to Check Logs, and Safe Rollback

Key Concepts
  • CI passing means the code compiles and tests pass; CD failing means the deployment broke, not the business logic.
  • Read the CD pipeline log first (the command that failed), then the cluster's view (rollout status, pod events, container logs).
  • ImagePullBackOff means a bad image tag or registry credential; CrashLoopBackOff means the app crashes at startup.
  • A failing readiness probe that never passes usually means an unreachable dependency or a wrong probe path.
  • Roll back to the last known-good version immediately using kubectl rollout undo, helm rollback, or git revert.
  • Preserve the failed pod's logs and describe output before it is recycled, because that evidence disappears.

Definition

CI passing but CD failing means the code is fine but the deployment broke, so read the CD log, then the cluster events and pod logs, and roll back to the last known-good version while preserving evidence.
  • CI passed – the code compiles, tests pass, and the artifact is built; the problem is not the business logic.
  • CD failed – the deployment itself broke: a bad manifest, a missing secret, an image pull failure, a failing probe, or insufficient resources.
  • Debug order – read the CD pipeline log first, then kubectl rollout status, kubectl describe pod (events), and kubectl logs --previous.
  • Rollback – return to the last known-good version using the platform's mechanism (kubectl rollout undo, helm rollback, or git revert).
  • Evidence preservation – capture the failed pod's logs and describe output before the pod is replaced.

Imagine the National Bank of India certified a batch of cheques as valid (CI passed) but the courier failed to deliver them (CD failed). The cheques are fine; the delivery broke. To debug, read the courier's trip log (CD pipeline log), then ask the destination branch what they saw (cluster events and pod logs): was the address wrong (bad manifest), the gate locked (missing secret), or the parcel rejected on inspection (failing probe)?

Understanding the Concepts

What does CI passing but CD failing mean?

CI already proved the code compiles and the tests pass, so the failure is almost never the business logic. CD failures are about getting the artifact running in an environment. The usual suspects are deployment configuration, missing secrets, image pull issues, failing probes, or cluster resource constraints.

TEXT
  CI: Build ✓  Test ✓  Scan ✓  Publish ✓
  CD: Deploy ✗

  The artifact is fine. The deployment is broken.

Where do you check the CD pipeline log?

Start with the CD job's console output. In Jenkins this is the build console; in GitHub Actions it is the job log. The log shows the exact command that failed and its stderr.

BASH
# Jenkins CD job console output (example):
[Deploy Prod] kubectl set image deployment/transfer-service ...
error: the server could not find the requested resource
# This tells you the Deployment resource does not exist in the namespace.

# GitHub Actions job log (example):
Run kubectl apply -f prod/
Error from server (Forbidden): deployments.apps is forbidden:
  User "ci-bot" cannot create resource "deployments" in namespace "prod"
# This tells you the service account lacks permissions.

How do you check the cluster's view?

After the pipeline log, ask the cluster directly. These four commands cover almost every CD failure.

BASH
# 1. Rollout status: did the new revision become available?
kubectl rollout status deployment/transfer-service -n prod
# "Waiting for deployment rollout to finish: 1 out of 3 new replicas available"

# 2. Pod states: what are the new pods doing?
kubectl get pods -n prod -l app=transfer-service
# NAME                               READY   STATUS             RESTARTS
# transfer-service-7d4c8-abc12       0/1     ImagePullBackOff   0
# transfer-service-7d4c8-def34       0/1     CrashLoopBackOff   3

# 3. Pod events: why is the pod failing?
kubectl describe pod transfer-service-7d4c8-abc12 -n prod
# Events:
#   Failed to pull image "registry.bank.local/transfer-service:wrong-tag":
#   manifest unknown

# 4. Container logs: what did the app say before crashing?
kubectl logs transfer-service-7d4c8-def34 -n prod --previous
# java.lang.IllegalStateException: Required key 'DB_PASSWORD' not found

How do you map symptoms to causes?

| Pod Status | Likely Cause | Fix |
|---|---|---|
| ImagePullBackOff | Wrong image tag, missing registry credentials, or tag does not exist | Fix the tag or registry secret |
| CrashLoopBackOff | App crashes at startup (missing config, bad property, OOM) | Check kubectl logs --previous |
| Pending | Cluster cannot schedule (insufficient CPU/memory, wrong node selector) | Check kubectl describe pod events |
| Running but not Ready | Readiness probe failing (wrong path, wrong port, unreachable dependency) | Fix the probe or the dependency |
| CreateContainerConfigError | Missing Secret or ConfigMap referenced by the pod spec | Create the missing resource |

BASH
# CrashLoopBackOff: the app needs DB_PASSWORD but it is missing.
kubectl logs transfer-service-pod -n prod --previous
# Output: "Required key 'DB_PASSWORD' not found in environment"

# Fix: create the missing secret.
kubectl create secret generic transfer-secrets \
  --from-literal=DB_PASSWORD='s3cur3p@ss' -n prod

How do you roll back safely?

Roll back to the last known-good version immediately. The rollback target is a version that was previously healthy in that same environment.

BASH
# Option 1: Kubernetes rollout undo
kubectl rollout undo deployment/transfer-service -n prod
# Returns to the previous ReplicaSet.

# Option 2: Helm rollback
helm rollback transfer-service 3 -n prod
# Returns to Helm release revision 3.

# Option 3: GitOps (git revert)
git revert HEAD   # Revert the commit that bumped the image tag
git push          # ArgoCD reconciles back to the previous version.
BASH
# Confirm health after rollback
kubectl rollout status deployment/transfer-service -n prod
# "deployment successfully rolled out"

curl -f https://transfer.bank.local/actuator/health
# {"status":"UP"}

What about database safety during rollback?

If the failed deploy ran a non-backward-compatible migration, rolling the code back is dangerous. This is why migrations must be backward-compatible (expand-and-contract).

SQL
-- WRONG: the migration dropped a column. Rolling back the code
-- means the old code reads a column that no longer exists.
ALTER TABLE transfer DROP COLUMN legacy_status;

-- CORRECT: expand-and-contract. Add the new column first,
-- deploy code that writes both, then drop the old column
-- only after the rollout is fully complete.
ALTER TABLE transfer ADD COLUMN transfer_status VARCHAR(20);

How do you preserve evidence?

Before the failed pods are replaced, capture their logs and describe output. Once the pods are gone, the diagnostic trail is gone.

BASH
# Capture evidence before rollback
kubectl logs transfer-service-pod -n prod --previous > /tmp/failed-pod.log
kubectl describe pod transfer-service-pod -n prod > /tmp/failed-pod-describe.txt

# Then roll back
kubectl rollout undo deployment/transfer-service -n prod

What should a senior engineer know?

  • A failed production deploy and its rollback are an auditable event in banking. Record what failed, what you rolled back to, and when.
  • Always confirm health after rollback with the same checks: rollout status, probes, and a smoke transfer.
  • The debug order is always pipeline log first, then cluster events, then container logs. Do not skip steps.
  • Keep migrations backward-compatible so rolling back the code is always safe, even if a migration ran.
BASH
# Senior pattern: full incident response
# 1. Capture evidence
kubectl logs pod --previous > evidence.log
kubectl describe pod > evidence-describe.txt

# 2. Roll back immediately
kubectl rollout undo deployment/transfer-service -n prod

# 3. Confirm health
kubectl rollout status deployment/transfer-service -n prod
curl -f https://transfer.bank.local/actuator/health

# 4. Record incident
echo "Incident: CD failed at $(date). Rolled back to previous revision." \
  >> /var/log/bank-incidents.log

Visual Overview

DIAGRAM
  Debug order:

  ┌──────────────────────────┐
  │  1. CD Pipeline Log       │
  │  Which step failed?       │
  │  What was the stderr?     │
  └────────────┬─────────────┘
               ▼
  ┌──────────────────────────┐
  │  2. kubectl rollout status│
  │  Did the revision deploy? │
  └────────────┬─────────────┘
               ▼
  ┌──────────────────────────┐
  │  3. kubectl get pods      │
  │  ImagePullBackOff?        │
  │  CrashLoopBackOff?        │
  │  Pending?                 │
  └────────────┬─────────────┘
               ▼
  ┌──────────────────────────┐
  │  4. kubectl describe pod  │
  │  Events section: reason   │
  └────────────┬─────────────┘
               ▼
  ┌──────────────────────────┐
  │  5. kubectl logs --previous│
  │  Application startup error│
  └────────────┬─────────────┘
               ▼
  ┌──────────────────────────┐
  │  6. Roll back safely      │
  │  kubectl rollout undo     │
  │  helm rollback            │
  │  git revert               │
  └──────────────────────────┘

Key Takeaway

When CI passes but CD fails, the first thing to do is reframe the problem. CI proved the code compiles and tests pass, so the failure is almost never the business logic. It is the deployment: a bad manifest, a missing secret, an image pull issue, a failing probe, or insufficient resources.

My debug order is pipeline log first, then cluster events outward. I read the CD console to see which step failed and its error. Then I check the cluster: rollout status, pod states, describe events, and the previous container logs. Each symptom maps to a specific cause, and the logs usually name it directly.

Rolling back safely means returning to the last version that was actually healthy in that environment. With Kubernetes that is rollout undo, with Helm that is helm rollback, and in GitOps that is a git revert. I always preserve the failed pod's logs and describe output before the pods are recycled, because that evidence disappears.

Remember one sentence: CI passing means the code is fine, so debug the deployment, not the business logic, and roll back to the last known-good version immediately.

---

Share
Use ← → arrow keys to navigate