Definition
When Pods are Running but get no traffic, the containers are healthy yet something between the client and the Pod is broken.
- Readiness – the readiness probe must pass so the Pod is marked Ready and included in the Service endpoints.
- Service selector – the selector must match the Pod labels; the EndpointSlice must list the Pod IPs.
- Port mapping – the Service port and targetPort must match the container's actual listening port.
- Ingress rules – the Ingress host and path must point at the right Service and port, and the controller must be running.
- NetworkPolicy – a default-deny policy can block traffic from the Ingress controller unless an explicit allow rule exists.
- LB health checks – the cloud load balancer's own health checks must consider the backend healthy.
At the National Bank of India, the teller is at the desk and ready to serve, but no customers arrive. The missing customers could be because the OPEN sign never flipped on (readiness), the branch directory lists the wrong desk number (Service selector), the reception sends visitors to the wrong floor (Ingress rules), a security door blocks the corridor (NetworkPolicy), or the building's front guard marked this branch closed on his sheet (load balancer health check). You check each checkpoint from the street inward.
Understanding the Concepts
How do you start the diagnosis?
The golden rule is to test from the inside out: curl the Pod IP directly, then the Service, then through the Ingress, then from outside. Each successful hop tells you the break is above it. Each failed hop tells you the break is at or below it.
# Start from inside: curl the Pod directly
kubectl run debug --rm -it --image=curlimages/curl -- \
curl -s http://10.0.1.5:8080/actuator/health
# If this works, the app is fine. The break is above the Pod.
# Next: curl the Service
kubectl run debug --rm -it --image=curlimages/curl -- \
curl -s http://transfer-service.prod.svc.cluster.local:80
# If this fails, the break is at the Service layer.
# Next: curl through the Ingress from inside
kubectl run debug --rm -it --image=curlimages/curl -- \
curl -s -H "Host: api.nbi.com" http://ingress-nginx-controller.ingress-nginx:80/transfers
# If this fails, the break is at the Ingress layer.
# Finally: curl from outside
curl -s https://api.nbi.com/transfers
Is the readiness probe passing?
A Running Pod whose readiness probe is failing is deliberately excluded from the Service. Check the READY column.
kubectl get pods -n prod
# NAME READY STATUS RESTARTS
# transfer-service-6d4f9c7b8a-x9k 0/1 Running 0 <- 0/1 means NOT ready
# transfer-service-6d4f9c7b8a-y2m 1/1 Running 0 <- ready
# Check why readiness is failing
kubectl describe pod transfer-service-6d4f9c7b8a-x9k -n prod
# Events:
# Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503
# Fix: resolve the underlying issue (DB connection, dependency down)
# The Pod will become Ready and rejoin the Service automatically
Does the Service selector match the Pod labels?
This is the single most common cause of Running Pods with no traffic. If the selector does not match the Pod labels, the EndpointSlice is empty.
# Check the Service selector
kubectl get svc transfer-service -n prod -o jsonpath='{.spec.selector}'
# {"app":"transfer"}
# Check the Pod labels
kubectl get pods -n prod --show-labels
# NAME LABELS
# transfer-service-6d4f9c7b8a-x9k app=transfer-service <- MISMATCH!
# The Service expects app=transfer but the Pods have app=transfer-service
# Result: empty endpoints, no traffic
kubectl get endpoints transfer-service -n prod
# NAME ENDPOINTS
# transfer-service <none> <- empty!
# WRONG - selector does not match Pod labels
apiVersion: v1
kind: Service
metadata:
name: transfer-service
spec:
selector:
app: transfer # Service looks for app=transfer
# But the Deployment creates Pods with app=transfer-service
# CORRECT - selector matches Pod labels exactly
apiVersion: v1
kind: Service
metadata:
name: transfer-service
spec:
selector:
app: transfer-service # matches Pod labels
Does the port mapping match?
The Service port is what clients call. The targetPort is what the container listens on. They must align.
# WRONG - targetPort does not match container port
apiVersion: v1
kind: Service
spec:
ports:
- port: 80
targetPort: 8081 # but the container listens on 8080
# CORRECT
apiVersion: v1
kind: Service
spec:
ports:
- port: 80
targetPort: 8080 # matches containerPort
# Verify the container's actual listening port
kubectl get pod transfer-service-x9k -n prod \
-o jsonpath='{.spec.containers[0].ports[0].containerPort}'
# 8080
Is the Ingress configured correctly?
The Ingress host and path rules must name the correct Service and port, and the Ingress controller must be running with an external address.
kubectl describe ingress banking-ingress -n prod
# Rules:
# Host Path Backend
# api.nbi.com /transfers transfer-service:80
# api.nbi.com /loans loan-service:80
# Check the Ingress has an external address
kubectl get ingress banking-ingress -n prod
# NAME CLASS HOSTS ADDRESS PORTS
# banking-ingress nginx api.nbi.com 203.0.113.50 80
# Common Ingress issues:
# - Wrong service name in the backend
# - Path does not match (Prefix vs Exact)
# - Missing IngressClass annotation
# - Ingress controller not running
kubectl get pods -n ingress-nginx
Is a NetworkPolicy blocking traffic?
In a locked-down banking namespace, a default-deny policy blocks all ingress unless an explicit allow rule permits traffic from the Ingress controller.
kubectl get networkpolicy -n prod
# NAME POD-SELECTOR AGE
# default-deny <none> 30d <- blocks all ingress
# No allow rule for the Ingress controller namespace!
# Fix: allow ingress from the Ingress controller namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-controller
namespace: prod
spec:
podSelector:
matchLabels:
app: transfer-service
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- protocol: TCP
port: 8080
Are the load balancer health checks passing?
The cloud load balancer runs its own health checks. If those checks return non-200, the LB marks the backend unhealthy and sends no traffic, even though Kubernetes thinks everything is fine.
# Check target group health in AWS (for ALB/NLB)
aws elbv2 describe-target-health --target-group-arn <arn>
# If targets are "unhealthy", check which health check path the LB is using
# Common issue: LB health check hits / but the app returns 404 on /
# Fix: set the LB health check path to the app's health endpoint
What are common mistakes?
The most common mistake is restarting Pods that were never broken. The Pods are Running and the application is healthy. The problem is in the networking layers between the client and the Pod.
# WRONG - restart Pods (they were never broken)
kubectl rollout restart deployment/transfer-service -n prod
# Problem persists because it was a label mismatch, not a Pod issue
# CORRECT - check endpoints first
kubectl get endpoints transfer-service -n prod
# <none> <- the real problem is here
What should a senior engineer know?
A senior engineer always checks endpoints early in the diagnosis. An empty endpoint list immediately narrows the problem to either a label mismatch or failing readiness, both of which are quick to fix.
The layered approach matters because several completely different faults produce the same symptom of Running Pods with no traffic. A methodical walk from the inside out finds the break faster and safer than trial and error.
# Quick diagnostic summary command
kubectl get pods,svc,endpoints,ingress,networkpolicy -n prod
Visual Overview
Traffic Path and Diagnostic Layers
====================================
Client (Priya's mobile app)
│
│ Layer 6: External Load Balancer health checks
▼
┌──────────────────────────┐
│ Cloud Load Balancer │ ← Is the target healthy?
└────────────┬─────────────┘
│
│ Layer 5: Ingress rules (host, path, backend)
▼
┌──────────────────────────┐
│ Ingress Controller │ ← Correct Service and port?
└────────────┬─────────────┘
│
│ Layer 4: NetworkPolicy (allow ingress?)
▼
┌──────────────────────────┐
│ NetworkPolicy │ ← Is traffic from controller allowed?
└────────────┬─────────────┘
│
│ Layer 3: Service selector + endpoints
▼
┌──────────────────────────┐
│ Service: transfer-svc │ ← Selector matches? Endpoints non-empty?
│ port → targetPort match? │
└────────────┬─────────────┘
│
│ Layer 2: Readiness probe
▼
┌──────────────────────────┐
│ Pod: readiness passing? │ ← READY 1/1 or 0/1?
└────────────┬─────────────┘
│
│ Layer 1: Application
▼
┌──────────────────────────┐
│ Container: listening on │ ← curl Pod IP:port works?
│ the expected port? │
└──────────────────────────┘
Key Takeaway
When Pods are Running but receive no traffic, the containers themselves are healthy and the fault lies somewhere between the client and the Pod. The golden rule is to test from the inside out: curl the Pod directly, then the Service, then through the Ingress, then from outside. Each successful hop tells you the break is above it.
Start with readiness, because a Running Pod whose readiness probe is failing is deliberately excluded from the Service. Next check the Service selector against Pod labels and look at the endpoint list, because an empty endpoint list is the most common cause. Then verify port mappings, Ingress rules, NetworkPolicy, and load balancer health checks.
Several completely different faults produce the exact same symptom, so a methodical walk is faster and safer than trial and error. At the National Bank of India, a refactor renamed the Pod label but left the Service selector pointing at the old label, so the endpoint list went empty and the mobile app timed out. Aligning the selector with the new label restored traffic instantly.
Remember one sentence: Running means alive, not reachable, so trace readiness, endpoints, Ingress, NetworkPolicy, and LB health layer by layer from the inside out.
---