Definition
When an EKS cluster is slow but CPU and memory look normal, the bottleneck is on a dimension those two metrics do not capture.
- Application – latency percentiles, error rate, thread pool saturation, connection pool saturation, downstream database latency.
- Kubernetes – scheduling delays, restarts, events, probe failures, Pending Pods.
- Node – disk I/O, network throughput, disk pressure, kernel issues, container runtime.
- Networking – CoreDNS/DNS latency, service-to-service calls, NetworkPolicy, load balancer health.
- Control plane – API server latency, etcd health, throttling.
- Why CPU/memory mislead – most latency comes from blocking on a resource (pool, DNS, downstream), not burning cycles.
At the National Bank of India, the branch is slow but every teller looks relaxed and the lights are not dimming. The real delay could be that there is only one phone line to the vault and everyone is queued on it (connection pool), the directory desk is slow to look up extensions (DNS), the central records office is slow to answer (downstream database), or the front-gate turnstile is jammed (load balancer). You would never find any of these by only watching how hard the tellers are working.
Understanding the Concepts
Why do CPU and memory mislead?
Most latency comes from blocking on a resource, not from burning CPU cycles. A request thread that is parked waiting for a database connection, a DNS answer, or a downstream response consumes almost no CPU while the user waits seconds. So the first discipline is to stop staring at the two easy dashboards.
# CPU and memory look normal
kubectl top pods -n prod
# NAME CPU MEMORY
# transfer-service-6d4f9c7b8a-x9k 120m 450Mi <- 12% CPU, normal memory
# transfer-service-6d4f9c7b8a-y2m 110m 440Mi
# But p99 latency is 3 seconds instead of 100ms
# CPU tells you nothing about where the 3 seconds are spent waiting
How do you troubleshoot the application layer?
Look at request latency percentiles, error rate, and pool saturation. A thread pool or connection pool that is exhausted will queue requests, causing latency with flat CPU.
# Check HikariCP connection pool metrics (Prometheus)
# hikaricp_connections_active / hikaricp_connections_max
# If active == max, requests are queuing for a connection
# Check downstream database query latency
# db_query_duration_seconds{quantile="0.99"}
# If p99 query latency spiked, the database is slow
# Distributed tracing (Jaeger/Zipkin) attributes time to each hop
# Trace shows:
# transfer-service: 50ms own time
# -> ledger-db query: 2800ms <- the slow hop!
# Total: 2850ms
# Check thread pool via Spring Boot Actuator
curl http://transfer-service:8080/actuator/metrics/executor.pool.size
curl http://transfer-service:8080/actuator/metrics/executor.queue.size
# If queue.size is growing, the thread pool is saturated
How do you troubleshoot the Kubernetes layer?
Check for scheduling delays, restarts, probe failures, and events. A Pod that keeps failing readiness is quietly removed from the Service and concentrates load on fewer Pods, raising latency.
# Check for Pods not ready (load concentrated on fewer Pods)
kubectl get pods -n prod
# NAME READY STATUS RESTARTS
# transfer-service-6d4f9c7b8a-x9k 1/1 Running 0
# transfer-service-6d4f9c7b8a-y2m 0/1 Running 0 <- NOT ready!
# transfer-service-6d4f9c7b8a-z3n 1/1 Running 0
# Only 2 of 3 Pods serving traffic -> each handles 50% more load
# Check for Pending Pods (not enough capacity)
kubectl get pods -n prod | grep Pending
# Check events for scheduling or probe issues
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20
How do you troubleshoot the node layer?
Disk I/O, network saturation, disk pressure, and container runtime issues can slow every Pod on a node even though CPU and memory read normal.
# Check for disk pressure or network issues on nodes
kubectl describe nodes | grep -E "DiskPressure|NetworkUnavailable|Conditions" -A5
# Check node-level metrics
# node_disk_io_time_seconds_total <- high I/O wait
# node_network_transmit_drop_total <- network drops
# node_filesystem_avail_bytes <- disk nearly full
# CPU throttling can cause latency even with normal average CPU
# container_cpu_cfs_throttled_periods_total > 0
# Throttled periods mean the container hit its CPU limit
How do you troubleshoot the networking layer?
DNS is the classic hidden cause. If CoreDNS is overloaded or misconfigured, every outbound call pays a DNS penalty that compounds across microservice hops.
# Check CoreDNS latency and error rate
kubectl top pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20
# Measure DNS resolution time from inside a Pod
kubectl exec -it transfer-service-x9k -- \
time nslookup ledger-db.prod.svc.cluster.local
# real 0m0.002s <- healthy
# real 0m0.500s <- DNS is slow!
# Check for NetworkPolicy blocking or causing retries
kubectl get networkpolicy -n prod
# Check load balancer health and connection limits
aws elbv2 describe-target-health --target-group-arn <arn>
How do you troubleshoot the control plane?
A slow API server or unhealthy etcd makes controllers, scheduling, and any in-cluster client slow.
# Check API server latency (CloudWatch for EKS)
# apiserver_request_duration_seconds{verb="GET",resource="pods"}
# Check for API server throttling
# apiserver_dropped_requests_total > 0
# Check etcd health (EKS managed, but observable)
# etcd_request_duration_seconds{type="range"}
What is the systematic approach?
Follow the request path, not a single dashboard. The five-layer approach ensures you find the real bottleneck.
# Layer 1: Application
curl -w "@curl-format.txt" http://transfer-service:8080/transfers/health
# Check latency, pool metrics, traces
# Layer 2: Kubernetes
kubectl get pods,events -n prod
# Check readiness, restarts, scheduling
# Layer 3: Node
kubectl describe nodes | grep Conditions -A5
# Check disk I/O, network, pressure, throttling
# Layer 4: Networking
kubectl exec -it debug -- nslookup transfer-service
# Check DNS, service-to-service, NetworkPolicy, LB
# Layer 5: Control plane
# Check API server latency, etcd, throttling
What are common mistakes?
The most common mistake is adding more Pods because the system is slow. If the bottleneck is a connection pool or DNS, more Pods waste money and fix nothing.
# WRONG - see slowness, scale up
kubectl scale deployment/transfer-service --replicas=10 -n prod
# Latency unchanged because the bottleneck is a 20-connection DB pool
# 10 Pods each competing for 20 connections makes it worse
# CORRECT - find the bottleneck first, then fix the right layer
# Trace shows: DB pool wait = 2.5s
# Fix: increase HikariCP pool size or optimize slow queries
What should a senior engineer know?
Distributed tracing is the most powerful diagnostic tool because it attributes the wait time to the exact hop that is slow. Without tracing, you are guessing across five layers.
CPU throttling (CFS throttled periods) can cause latency even when average CPU looks normal. A container that bursts to its CPU limit gets throttled for the remainder of the period, which adds latency spikes.
# Check for CPU throttling (often missed)
kubectl exec transfer-service-x9k -- cat /sys/fs/cgroup/cpu/cpu.stat
# throttled_time > 0 <- container is being throttled
Visual Overview
Five-Layer Troubleshooting Path
=================================
Symptom: EKS cluster is slow, CPU and memory normal
│
▼
┌─────────────────────────────────────────────────┐
│ Layer 1: Application │
│ - p99 latency, error rate │
│ - Thread pool: queue size, active count │
│ - Connection pool: active vs max, wait time │
│ - Downstream DB: query latency │
│ - Distributed tracing: which hop is slow? │
└────────────────────┬────────────────────────────┘
▼
┌─────────────────────────────────────────────────┐
│ Layer 2: Kubernetes │
│ - Pods not ready (load concentrated) │
│ - Pending Pods, scheduling delays │
│ - Restarts, probe failures, events │
└────────────────────┬────────────────────────────┘
▼
┌─────────────────────────────────────────────────┐
│ Layer 3: Node │
│ - Disk I/O, disk pressure │
│ - Network saturation, drops │
│ - CPU throttling (CFS) │
└────────────────────┬────────────────────────────┘
▼
┌─────────────────────────────────────────────────┐
│ Layer 4: Networking │
│ - CoreDNS latency and errors │
│ - Service-to-service call latency │
│ - NetworkPolicy blocking/retries │
│ - Load balancer health and limits │
└────────────────────┬────────────────────────────┘
▼
┌─────────────────────────────────────────────────┐
│ Layer 5: Control Plane │
│ - API server request latency │
│ - etcd health and latency │
│ - Throttling / dropped requests │
└─────────────────────────────────────────────────┘
Key Takeaway
When a cluster is slow but CPU and memory look normal, the system is waiting, not computing. A request thread parked on a saturated connection pool, a slow DNS lookup, or a slow downstream consumes almost no CPU while the user waits seconds. CPU and memory are just two signals among a dozen.
Troubleshoot the whole request path across five layers. At the application layer, check latency percentiles, pool saturation, and downstream database latency, ideally with distributed tracing. At the Kubernetes layer, check for Pods failing readiness that concentrate load. At the node layer, check disk I/O, network saturation, and CPU throttling. At the networking layer, check CoreDNS latency, which adds a penalty to every outbound call. At the control plane layer, check API server and etcd latency.
The wrong instinct is to add more Pods because the system is slow. If the bottleneck is a connection pool or DNS, more Pods waste money and fix nothing. Find the bottleneck first, then fix the right layer.
Remember one sentence: slow with normal CPU means the system is waiting, so trace latency across application pools, Kubernetes readiness, node I/O, DNS, and the control plane.
---