Definition
- End-to-end debugging – Tracing the request along the entire path it travels, checking each boundary in order until you find the broken link.
- DNS – Verify the hostname resolves to the current load balancer address, not a stale cached record.
- Load balancer and ingress – Confirm the ALB target group is healthy and the Kubernetes Ingress maps the host and path to the correct Service.
- Service and Pod readiness – Check that the Service has ready Pod IPs in its endpoints list; a failing readiness probe silently removes Pods.
- Auth service – Verify the authentication service is returning 200, not 500 from an exhausted pool or 401 from a rotated signing key.
- Logs and traces – Use structured logs filtered by account ID and distributed traces keyed on a correlation ID to pinpoint the exact failing span.
Imagine Rajesh at the National Bank of India types his correct password but still cannot get in. You retrace his journey like a detective: did his request even find the bank's address (DNS)? Did the front gate let him in (load balancer and ingress)? Was any teller counter actually open (Service and Pod readiness)? Did the ID-check desk verify him (auth service)? And what do the bank's CCTV logs and journey maps say (logs and traces)? You check each checkpoint in order until you find the one that is closed.
Understanding the Concepts
What is end-to-end debugging for access failures?
End-to-end debugging means systematically tracing a user's request along every hop it travels, from DNS resolution through to the application layer, confirming at each boundary whether the request arrived and what response it received. The goal is to isolate the one layer that is failing instead of guessing. You work from the outside in because an outer failure makes every inner layer unreachable, so confirming outer layers first eliminates large categories of causes quickly.
# The debugging order (outside in)
# 1. DNS resolution
# 2. Load balancer / Ingress health
# 3. Service and Pod readiness
# 4. Auth service health
# 5. Logs and distributed traces
How do you check DNS resolution?
Start at DNS because if the hostname does not resolve, nothing else matters. A common failure is a DNS record pointing at a deleted or old load balancer, so the browser shows a cached login page but new requests go nowhere.
# Check DNS resolution for the banking API
nslookup api.nationalbank.in
# Or with more detail:
dig api.nationalbank.in +short
# Expected: returns the current ALB IP, e.g., 10.0.1.50
# Verify it matches the actual load balancer
aws elbv2 describe-load-balancers \
--names banking-alb \
--query 'LoadBalancers[0].DNSName'
# Compare: the dig result should point to this ALB DNS
# If nslookup returns an old IP or NXDOMAIN:
# Root cause: stale DNS record or deleted load balancer
# Fix: update the DNS record in Route 53 to the current ALB
How do you check load balancer and ingress health?
On AWS, check the ALB target group health. In Kubernetes, inspect the Ingress resource to confirm it has an address and the backend rules map the host and path to the correct Service.
# AWS: Check ALB target group health
aws elbv2 describe-target-health \
--target-group-arn arn:aws:elasticloadbalancing:ap-south-1:123456:targetgroup/banking-tg/abc123
# Expected output for healthy targets:
{
"TargetHealthDescriptions": [
{
"Target": { "Id": "i-0abc123", "Port": 8080 },
"TargetHealth": { "State": "healthy" }
}
]
}
# If State is "unhealthy" or "draining", traffic is not reaching the app.
# Kubernetes: Inspect the Ingress
kubectl describe ingress banking-ingress -n banking
# Check: does it have an ADDRESS? Do the rules map the correct host and path?
# A frequent cause is an ingress rule pointing at a Service name that was renamed.
How do you verify Service and Pod readiness?
This is where access failures most often hide. A Pod that is Running but not Ready (readiness probe failing) is silently removed from the Service endpoints, which perfectly explains "my password is right but the app will not load."
# Check if the Service has any ready endpoints
kubectl get endpoints banking-service -n banking
# Expected: Pod IPs listed
NAME ENDPOINTS AGE
banking-service 10.244.1.5:8080,10.244.2.6:8080 30d
# If ENDPOINTS is empty: the Service routes to nothing!
# Even though Pods may show as Running.
# Check Pod status and readiness
kubectl get pods -l app=banking-api -n banking
# Look for: Running but 0/1 READY, CrashLoopBackOff, or OOMKilled
# Describe a suspect Pod for events and probe status
kubectl describe pod banking-api-xyz -n banking
# Look for: Readiness probe failed, Back-off restarting
# Check Pod logs for startup errors
kubectl logs banking-api-xyz -n banking --tail=100
# Common finding:
# Pod is Running (container started) but READY is 0/1
# because the readiness probe to /health returns 503
# since the database connection pool failed to initialize.
# The Pod is alive but the Service dropped it from endpoints.
How do you check the auth service?
Correct credentials do not mean the auth service is up. It may be returning 500 because its database connection pool is exhausted, or returning 401 because a clock skew invalidated JWT signature validation, or because the token-signing key was rotated and the verifying service still has the old public key.
# Hit the auth service health endpoint directly
curl -s http://auth-service.banking.svc.cluster.local:8080/health
# Expected: {"status": "UP"}
# Check auth service logs for the specific user
kubectl logs -l app=auth-service -n banking --tail=200 | grep "SAV2001"
# Look for: token validation errors, connection pool exhausted, 500 responses
# Example auth service log showing the real cause:
# ERROR TokenValidator - JWT signature validation failed for user SAV2001:
# PublicKey mismatch. Expected key ID: key-v2, got: key-v1
# Root cause: signing key was rotated but the verifier has the old key cached.
How do you use logs and distributed traces to pinpoint the failure?
A correlation ID or trace ID attached to the request lets you follow it across gateway, auth, and application services. The span that errors or never starts tells you precisely where the request died.
# Search structured logs by customer account ID
kubectl logs -l app=banking-api -n banking --tail=500 | \
grep '"accountId":"SAV2001"'
# In a centralized logging system (ELK, Loki):
# Query: accountId="SAV2001" AND level="ERROR" AND timestamp > "2024-01-15T10:00:00"
# In a tracing backend (Jaeger):
# Search by tag: user.accountId=SAV2001
# Find the trace, expand spans, look for the red (errored) span
# Example trace showing the failure:
# Span 1: ingress-gateway 200ms OK
# Span 2: auth-service 50ms OK
# Span 3: banking-api ERROR Connection refused to database
# -> Root cause: database is unreachable from the banking-api Pod
What are common mistakes when debugging access failures?
The biggest mistake is jumping to conclusions instead of checking each hop in order.
# WRONG: Immediately blaming the auth service without checking DNS
# "Credentials are correct, so it must be an auth bug."
# You restart the auth service, but the real problem is
# a stale DNS record pointing to a deleted load balancer.
# Hours wasted.
# CORRECT: Check DNS first, then LB, then Service, then auth
nslookup api.nationalbank.in # Step 1: DNS
curl -s -o /dev/null -w "%{http_code}" https://api.nationalbank.in/health # Step 2: LB
kubectl get endpoints banking-service # Step 3: Service
curl -s http://auth-service:8080/health # Step 4: Auth
# Each step confirms or eliminates a layer before moving inward.
What should a senior engineer know?
A senior engineer knows that the most common root causes for "correct credentials but cannot access" are: empty Service endpoints due to failing readiness probes, stale DNS records, ingress rules pointing at renamed Services, auth service key rotation mismatches, and clock skew invalidating JWT tokens. They also know to check for network policies that may be blocking traffic between namespaces, and to verify that TLS certificates have not expired, which can cause silent connection failures that look like access denials.
# Check for network policy blocking traffic
kubectl get networkpolicy -n banking
# Review: does any policy deny ingress to the banking-api Pods?
# Check TLS certificate expiry
echo | openssl s_client -connect api.nationalbank.in:443 2>/dev/null | \
openssl x509 -noout -dates
# If "notAfter" is in the past, the certificate has expired.
What are common interview follow-up questions?
- "What if DNS is fine but the app is still unreachable?" – Check the load balancer target health and ingress rules next, then Service endpoints.
- "How do you distinguish an auth failure from an infrastructure failure?" – An auth failure returns 401 or 403 with a body; an infrastructure failure returns 502, 503, or a connection timeout.
- "What is the fastest way to restore service while debugging?" – Roll back the most recent change (deploy, config, or infra) and verify the customer can access again before doing deep analysis.
Visual Overview
End-to-End Debugging Path (Outside In)
=======================================
Customer (Rajesh, SAV2001)
│
▼
┌─────────────┐
│ DNS │ nslookup / dig
│ (Route 53) │ Does hostname resolve to current LB?
└──────┬──────┘
│
▼
┌─────────────────┐
│ Load Balancer │ aws elbv2 describe-target-health
│ / Ingress │ Are targets healthy? Rules correct?
└──────┬──────────┘
│
▼
┌─────────────────┐
│ K8s Service │ kubectl get endpoints
│ + Pod Ready │ Does Service have ready Pod IPs?
└──────┬──────────┘
│
▼
┌─────────────────┐
│ Auth Service │ curl /health + check logs
│ (JWT / keys) │ Is it returning 200 or 500/401?
└──────┬──────────┘
│
▼
┌─────────────────┐
│ Logs & Traces │ grep accountId + Jaeger spans
│ (correlation) │ Which span errored or timed out?
└─────────────────┘
Rule: The FIRST hop that fails is your culprit.
Fix it, then recheck downstream.
Key Takeaway
When a customer reports they cannot access the application even though their credentials are correct, debug end-to-end by following the request along its entire path and checking each hop in a fixed order from the outside in. Start at DNS and confirm the hostname resolves to the current load balancer address. Then check load balancer and ingress health to confirm targets are healthy and routing rules are correct. Then verify Service and Pod readiness, which is where these issues most often hide: a Pod that is Running but not Ready because its readiness probe is failing gets silently removed from the Service endpoints, perfectly explaining why the login page loads from cache but every real request fails.
Only after confirming infrastructure do you check the auth service itself, because correct credentials do not guarantee the auth service is healthy; it might be returning 500 from an exhausted connection pool or rejecting valid tokens because a signing key was rotated and the verifier still holds the old public key. Finally, use structured logs filtered by the customer's account ID and distributed traces keyed on a correlation ID to pinpoint the exact failing span. At the National Bank of India, when Rajesh with account SAV2001 could not log in, this exact method took the team from a vague complaint to a precise root cause: DNS and ingress were healthy, but the banking Service had zero ready endpoints because the Pods' readiness probes were failing against an unreachable database.
Remember one sentence: debug access failures from the outside in, confirm at every boundary whether the request arrived and what status came back, and the first hop that fails is the broken link.
---