Why can Kubernetes autoscaling make an incident WORSE when the real bottleneck is downstream? | Microservices

Definition

Autoscaling the application tier against a saturated downstream adds more callers to the queue at the real bottleneck, and the idle-waiting replicas report low CPU that tricks the autoscaler into scaling further.

Hidden assumption -- HPA assumes the application is CPU-bound; more CPU equals more throughput.
Downstream bottleneck -- when PostgreSQL, Redis, or an external API has fixed capacity, throughput is set by the dependency, not the app.
Contention amplification -- more pods equals more connection pools equals more concurrent callers overwhelming the already-saturated resource.
Feedback loop -- pods blocked on I/O show low CPU, so the autoscaler thinks there is headroom and may scale further.
Retry amplification -- more pods times retries per pod equals multiplicative load on the struggling dependency.
Fix -- scale on the right signal; cap total connections with PgBouncer; add concurrency limits and retry budgets.

National Bank of India has one vault clerk who can hand out one file per minute. During a rush the manager hires 40 more tellers. But there is still one vault clerk. Now 60 tellers crowd the vault, the clerk is overwhelmed, every teller stands idle waiting (low CPU), and the manager hires even more. The one clerk was always the limit; adding tellers made the jam worse.

Understanding the Concepts

Why does HPA make it worse when the bottleneck is downstream?

The Horizontal Pod Autoscaler adds pods when CPU crosses a threshold, assuming the app tier is the bottleneck. When the real bottleneck is a downstream resource with fixed capacity (PostgreSQL max connections, Redis memory, external API rate limit), more pods add more callers to the already-saturated resource. Throughput drops, not rises.

JAVA

package com.bank.infra;

public class DownstreamBottleneck {
    public static void main(String[] args) {
        int dbMaxConnections = 100;
        int connectionsPerPod = 10;
        int pods = 10;
        int totalConnections = pods * connectionsPerPod;

        System.out.println("DB max connections: " + dbMaxConnections);
        System.out.println(pods + " pods x " + connectionsPerPod + " pool = " + totalConnections);
        System.out.println("Exactly at ceiling.");

        int scaledPods = 20;
        int scaledConnections = scaledPods * connectionsPerPod;
        System.out.println("\nHPA scales to " + scaledPods + " pods");
        System.out.println(scaledPods + " x " + connectionsPerPod + " = " + scaledConnections + " wanted");
        System.out.println("DB max: " + dbMaxConnections + " --> 100 connections starve");
        System.out.println("Throughput DROPS, not rises.");
    }
}

TEXT

Expected output:
DB max connections: 100
10 pods x 10 pool = 100
Exactly at ceiling.

HPA scales to 20 pods
20 x 10 = 200 wanted
DB max: 100 --> 100 connections starve
Throughput DROPS, not rises.

Why does the feedback loop mislead the autoscaler?

Pods blocked waiting on a slow database are parked on I/O, not burning CPU. Average CPU drops. The HPA sees low CPU and concludes there is headroom, so it adds more pods, which add more callers, which block more, which lower CPU further.

JAVA

package com.bank.infra;

public class MisleadingFeedback {
    public static void main(String[] args) {
        System.out.println("Pod CPU: 15% (parked on I/O, not busy!)");
        System.out.println("HPA thinks: 'plenty of headroom'");
        System.out.println("HPA action: add more pods");
        System.out.println("Result: MORE callers at the bottleneck");
        System.out.println("CPU drops further (more I/O wait)");
        System.out.println("HPA adds EVEN MORE pods --> death spiral");
    }
}

TEXT

Expected output:
Pod CPU: 15% (parked on I/O, not busy!)
HPA thinks: 'plenty of headroom'
HPA action: add more pods
Result: MORE callers at the bottleneck
CPU drops further (more I/O wait)
HPA adds EVEN MORE pods --> death spiral

How do retries amplify the problem?

Each pod has its own retry policy. When all pods retry failed requests, the dependency sees multiplicative load.

JAVA

package com.bank.infra;

public class RetryAmplification {
    public static void main(String[] args) {
        int pods = 20;
        int retriesPerPod = 3;
        int originalRequests = 1000;
        int totalWithRetries = originalRequests + (originalRequests * retriesPerPod);

        System.out.println("Original requests: " + originalRequests);
        System.out.println("Each pod retries failed requests " + retriesPerPod + " times");
        System.out.println("DB sees up to: " + totalWithRetries + " attempts");
        System.out.println("Struggling DB now has 4x more load from retries");
    }
}

TEXT

Expected output:
Original requests: 1000
Each pod retries failed requests 3 times
DB sees up to: 4000 attempts
Struggling DB now has 4x more load from retries

What is the correct autoscaling signal?

Scale on a metric that reflects the actual constraint, not blind CPU.

| Problem | Fix |
|---|---|
| Wrong signal | Autoscale on latency, queue depth, or connection-pool saturation |
| Connection overload | PgBouncer caps total connections regardless of pod count |
| Retry amplification | Retry budgets and concurrency limits |
| During incident | Scale DOWN or shed load, not scale up |

YAML

# WRONG: scale on CPU when bottleneck is downstream
metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70

# CORRECT: scale on a signal that reflects the actual constraint
metrics:
  - type: Pods
    pods:
      metric:
        name: http_server_requests_seconds_p99
      target:
        type: AverageValue
        averageValue: '200m'

JAVA

package com.bank.infra;

public class CorrectScalingSignal {
    public static void main(String[] args) {
        System.out.println("WRONG signal: CPU utilization (misleading for I/O-bound)");
        System.out.println("CORRECT signals:");
        System.out.println("  - p99 latency (reflects actual user experience)");
        System.out.println("  - queue depth (reflects backlog)");
        System.out.println("  - connection-pool saturation (reflects DB pressure)");
    }
}

TEXT

Expected output:
WRONG signal: CPU utilization (misleading for I/O-bound)
CORRECT signals:
  - p99 latency (reflects actual user experience)
  - queue depth (reflects backlog)
  - connection-pool saturation (reflects DB pressure)

How does PgBouncer fix connection overload?

PgBouncer sits between pods and PostgreSQL. No matter how many pods exist, PgBouncer limits the total connections to the DB's safe ceiling.

JAVA

package com.bank.infra;

public class PgBouncerSolution {
    public static void main(String[] args) {
        int pods = 20;
        int connectionsPerPod = 10;
        int pgBouncerMaxToDb = 80;

        System.out.println("Without PgBouncer:");
        System.out.println("  " + pods + " pods x " + connectionsPerPod
            + " = " + (pods * connectionsPerPod) + " connections --> DB overload");

        System.out.println("With PgBouncer:");
        System.out.println("  " + pods + " pods connect to PgBouncer");
        System.out.println("  PgBouncer max to DB: " + pgBouncerMaxToDb);
        System.out.println("  DB always sees <= " + pgBouncerMaxToDb + " connections");
        System.out.println("  Regardless of pod count");
    }
}

TEXT

Expected output:
Without PgBouncer:
  20 pods x 10 = 200 connections --> DB overload
With PgBouncer:
  20 pods connect to PgBouncer
  PgBouncer max to DB: 80
  DB always sees <= 80 connections
  Regardless of pod count

What is the diagnostic pattern?

When you see latency high, CPU low, and throughput flat or falling as pods increase, the bottleneck is downstream. Do not scale the app tier.

JAVA

package com.bank.infra;

public class DiagnosticPattern {
    public static void main(String[] args) {
        double latencyP99Ms = 2000;
        double cpuPercent = 15;
        double throughputQps = 200; // flat despite scaling

        System.out.println("Latency:    " + latencyP99Ms + " ms (HIGH)");
        System.out.println("CPU:        " + cpuPercent + "% (LOW)");
        System.out.println("Throughput: " + throughputQps + " QPS (FLAT)");

        if (latencyP99Ms > 1000 && cpuPercent < 30) {
            System.out.println("DIAGNOSIS: downstream bottleneck");
            System.out.println("ACTION: do NOT scale app tier");
            System.out.println("Instead: find saturated downstream, cap connections");
        }
    }
}

TEXT

Expected output:
Latency:    2000.0 ms (HIGH)
CPU:        15.0% (LOW)
Throughput: 200.0 QPS (FLAT)
DIAGNOSIS: downstream bottleneck
ACTION: do NOT scale app tier
Instead: find saturated downstream, cap connections

What are common mistakes?

The most common mistake is scaling up during an incident when the correct action is to scale down or shed load.

JAVA

package com.bank.infra;

public class CommonMistakes {
    public static void main(String[] args) {
        // WRONG: scaling up when bottleneck is downstream
        System.out.println("WRONG: latency high, CPU low --> HPA adds pods");
        System.out.println("More pods = more connections = more contention = WORSE");

        // CORRECT: diagnose first, then act
        System.out.println("CORRECT: check what pods are WAITING on");
        System.out.println("If downstream saturated: cap connections, shed load");
    }
}

TEXT

Expected output:
WRONG: latency high, CPU low --> HPA adds pods
More pods = more connections = more contention = WORSE
CORRECT: check what pods are WAITING on
If downstream saturated: cap connections, shed load

What should a senior engineer know?

When latency is high but CPU is low, scaling the application is almost always wrong.
Diagnose before acting: check what the application is waiting on.
Use PgBouncer or a shared connection gate so the DB never sees more than its safe connection count.
During an incident with a downstream bottleneck, the correct action is often to scale down or shed load.
Configure HPA with custom metrics (latency, queue depth) instead of blind CPU.

JAVA

package com.bank.infra;

public class SeniorInsights {
    public static void main(String[] args) {
        System.out.println("1. High latency + low CPU = downstream bottleneck");
        System.out.println("2. Diagnose BEFORE scaling: what is the pod waiting on?");
        System.out.println("3. PgBouncer: fixed connection count to DB");
        System.out.println("4. During incident: scale DOWN or shed load");
        System.out.println("5. HPA custom metrics: latency, queue depth, not CPU");
    }
}

TEXT

Expected output:
1. High latency + low CPU = downstream bottleneck
2. Diagnose BEFORE scaling: what is the pod waiting on?
3. PgBouncer: fixed connection count to DB
4. During incident: scale DOWN or shed load
5. HPA custom metrics: latency, queue depth, not CPU

Visual Overview

DIAGRAM

  HPA Feedback Loop (the trap)

  DB saturated --> pods wait on I/O --> CPU drops
       ▲                                    │
       │                                    ▼
  More pods  <-- HPA sees low CPU --> "add pods!"
       │                                    │
       └──── more connections ──────────────┘
       MORE callers at bottleneck = WORSE

  Correct Architecture
  ┌──────┐  ┌──────┐  ┌──────┐
  │Pod-1 │  │Pod-2 │  │Pod-N │
  └──┬───┘  └──┬───┘  └──┬───┘
     └─────────┼─────────┘
               ▼
        ┌──────────────┐
        │  PgBouncer   │  max 80 connections to DB
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │  PostgreSQL  │  sees <= 80 regardless of pod count
        └──────────────┘

Key Takeaway

Kubernetes HPA adds pods when CPU crosses a threshold, assuming the app tier is the bottleneck. When the real bottleneck is a downstream resource with fixed capacity, more pods add more callers, connections, and retries against the already-saturated dependency. Throughput drops, not rises. The feedback loop makes it worse: pods blocked on I/O show low CPU, which tricks the autoscaler into scaling further.

The fix is to scale on a signal that reflects the actual constraint (latency, queue depth, connection-pool saturation) and to cap the total load on the downstream regardless of pod count. PgBouncer ensures the database sees a fixed connection count whether you run 3 pods or 30.

At National Bank of India, the month-end statement spike used to collapse under autoscaling until Pankaj put PgBouncer in front of PostgreSQL so the database never saw more than 80 connections. He switched the HPA metric from CPU to p99 latency and added a retry budget so scaled-up pods did not multiply the offered load.

Remember one sentence: when latency is high but CPU is low, do not scale the tier in front of the bottleneck -- find the saturated downstream, cap connections with PgBouncer, and scale on a signal that reflects the real constraint.

---