Definition
- CPU bottleneck – per-request work (JWT verification, rate limiting, logging) multiplied by total traffic saturates the gateway's CPU.
- Single point of failure – all traffic funnels through the gateway, so if it slows or dies, the entire system appears down.
- Thread blocking – synchronous I/O in the request handler pins threads; under load, all threads block and the gateway stops accepting requests.
- Guiding rule – do as little work as possible on the request path: move non-critical work off it, cache expensive computations, never block.
- Horizontal scaling – necessary but not sufficient; making each request cheap is equally important.
Imagine every customer of National Bank of India must pass through one security desk. If that desk does heavy work per person, checking ID by phone call, stamping forms by hand, writing a logbook entry, then during salary week the line explodes. The small delay per person multiplied by thousands becomes a massive jam. And if that desk closes, nobody gets into the bank. The fix is to make the desk do the bare minimum per person.
Understanding the Concepts
Why does the gateway become a CPU bottleneck?
The gateway sits on the path of every request. Each request typically pays for JSON parsing, JWT signature verification (cryptographic work), a rate-limit check (Redis round trip), a log write, request/response transformation, and a new upstream connection. At 50,000 requests per second, small costs saturate CPU.
// Per-request costs at the gateway: // 1. TLS termination: ~0.1ms (if new connection) // 2. JSON parsing: ~0.05ms // 3. JWT verification: ~0.2ms (RSA signature check) // 4. Rate-limit check: ~0.5ms (Redis round trip) // 5. Synchronous logging: ~0.3ms (disk I/O) // 6. Route matching: ~0.01ms // 7. Upstream connection: ~1ms (new TCP+TLS handshake) // Total per request: ~2.16ms // At 50,000 req/s: 50,000 * 2.16ms = 108,000ms of work per second // That requires ~108 CPU-seconds per second → need 108+ cores!
Why is thread blocking the most dangerous failure?
If the gateway does synchronous blocking I/O, each in-flight request holds a thread. Under load, all threads block and the gateway stops accepting new requests.
// WRONG: synchronous blocking in the request handler
public class BlockingGatewayFilter {
public void filter(Request request) {
// Blocks a thread waiting for Redis
boolean allowed = redisClient.checkRateLimit(request.clientId());
// Blocks a thread waiting for auth service
Claims claims = authClient.validateToken(request.token());
// Blocks a thread waiting for disk
accessLogger.logSync(request);
// Forward to backend (blocks waiting for response)
Response response = backendClient.forward(request);
}
}
// Under load: 200 threads in the pool // Each request blocks for ~10ms (Redis + auth + logging) // Throughput: 200 / 10ms = 20,000 req/s max // At 50,000 req/s: 30,000 requests queued, waiting for threads // Eventually: thread pool exhausted → gateway returns 503 to everyone // Backend services are perfectly healthy but unreachable!
What is the guiding rule?
Do as little work as possible on the request path. This has three concrete parts.
// Part 1: Move non-critical work OFF the hot path
// WRONG: synchronous log write per request
accessLogger.logSync(request); // blocks thread for disk I/O
// CORRECT: async batched logging
accessLogger.logAsync(request); // enqueue, background worker flushes
// Request pays ~0.001ms (enqueue) instead of ~0.3ms (disk I/O)
// Part 2: Cache aggressively
// WRONG: fetch JWT public key from auth server per request
PublicKey key = authServer.fetchPublicKey(); // network call per request!
// CORRECT: cache the public key, refresh on rotation
PublicKey key = jwksCache.getKey(keyId); // local memory lookup
// JWKS cache refreshes every 5 minutes or when unknown keyId appears
// Per-request cost: ~0.001ms instead of ~5ms
// Part 3: Never do synchronous blocking I/O
// WRONG: blocking RestTemplate for upstream calls
Response response = restTemplate.postForObject(url, body, Response.class);
// CORRECT: non-blocking WebClient with timeout
Mono<Response> response = webClient.post()
.uri(url)
.bodyValue(body)
.retrieve()
.bodyToMono(Response.class)
.timeout(Duration.ofMillis(500));
// No thread is blocked; reactive event loop handles the I/O
How does caching cut per-request cost?
Anything expensive and repeatable should be cached so the gateway computes it once instead of per request.
// What to cache at the gateway:
// 1. JWT public keys (JWKS): refresh every 5 min, check locally per request
// 2. Rate-limit state: local token bucket, sync with Redis every 100ms
// 3. Route table: cached, rebuild only on RefreshRoutesEvent
// 4. Authorization decisions: short TTL if using OPA
// 5. TLS sessions: session resumption avoids re-handshake
// Before caching: JWT verification = 5ms (network) + 0.2ms (crypto)
// After caching: JWT verification = 0.2ms (crypto only)
// Savings: 96% reduction in JWT cost
Why is horizontal scaling necessary but not sufficient?
Running multiple gateway replicas adds capacity and removes the literal single point of failure, but if each replica wastes per-request work, you are buying hardware to do waste.
// Without optimization: // 1 replica: handles 20,000 req/s (threads block) // Need 50,000 req/s → add 3 replicas = $$$ // With optimization (non-blocking, cached, async logging): // 1 replica: handles 100,000 req/s // Need 50,000 req/s → 1 replica is enough, add 1 for redundancy // Combine: make each request cheap AND scale horizontally
How do you protect the gateway itself?
The gateway needs its own health checks, circuit breakers on downstream calls, and load shedding to degrade gracefully.
// Gateway self-protection:
// 1. Health check: /actuator/health → load balancer removes unhealthy replicas
// 2. Circuit breaker per downstream route → fail fast, not hang
// 3. Load shedding: when CPU > 80%, start rejecting lowest-priority requests
// 4. Connection limits: max connections per downstream service
// 5. Timeout on every downstream call: never wait forever
What are common mistakes? (wrong vs correct)
// WRONG: fetching JWT signing key over the network PER REQUEST
// 50,000 req/s * 5ms per network call = 250,000ms of network I/O
// CORRECT: fetch once, cache in memory, refresh every 5 minutes
// 50,000 req/s * 0.001ms per cache lookup = 50ms total
// WRONG: synchronous disk logging per request
// CORRECT: async batch logging (enqueue, flush every 100ms)
// WRONG: new TCP+TLS connection per upstream request
// CORRECT: connection pool with keep-alive, >99% reuse
What should a senior engineer know?
A senior engineer profiles where the CPU actually goes, because the biggest cost is often surprising (TLS handshakes and JWT verification, not business logic). They measure, optimize, and then scale.
// Senior optimization priority:
// 1. Profile: where does CPU time actually go?
// 2. Eliminate: remove unnecessary per-request work
// 3. Cache: anything expensive and repeatable
// 4. Async: move non-critical work off the hot path
// 5. Non-blocking: never block a thread on I/O
// 6. Pool: reuse connections, >99% reuse target
// 7. Scale: horizontal replicas behind load balancer
// 8. Protect: health checks, circuit breakers, load shedding
Visual Overview
Gateway Bottleneck
──────────────────
50,000 requests/second
│
▼
┌────────────────────────────┐
│ API Gateway (1 replica) │
│ │
│ Per request: │
│ TLS + JSON + JWT + Redis │
│ + Log + Transform + Conn │
│ = ~2ms per request │
│ │
│ 50K * 2ms = 100 CPU-sec/s │
│ → CPU saturated! │
└────────────────────────────┘
After Optimization
──────────────────
50,000 requests/second
│
▼
┌────────────────────────────┐
│ API Gateway (optimized) │
│ │
│ Cached JWT keys: -4.8ms │
│ Async logging: -0.3ms │
│ Connection pool: -0.9ms │
│ Local rate limit: -0.4ms │
│ = ~0.2ms per request │
│ │
│ 50K * 0.2ms = 10 CPU-sec/s│
│ → 10x headroom! │
└────────────────────────────┘
Key Takeaway
An API Gateway becomes the system's CPU bottleneck because every request passes through it, so whatever work it does per request is multiplied by the total traffic. At 50,000 requests per second, even small per-request costs like JWT verification, rate-limit checks, and synchronous logging add up to saturated CPU. And because all traffic funnels through it, if the gateway slows or dies, the entire system appears down even though the backend services are healthy.
The worst failure mode is thread blocking. At National Bank of India, the early gateway fetched the JWT signing key over the network on every request, logged synchronously, and opened a new upstream connection per request. During salary week, CPU hit 100%, threads piled up waiting on I/O, and the gateway stopped serving. The fix was the guiding rule: do as little work as possible on the request path. Cache the JWT keys locally, log asynchronously in batches, keep a connection pool with 99%+ reuse, and never block a thread on I/O.
Horizontal scaling is necessary but not sufficient. Running multiple replicas removes the single point of failure and adds capacity, but if each replica wastes per-request work, you are just paying for hardware to do the same waste. Combine making each request cheap with scaling out, and protect the gateway itself with health checks, circuit breakers, and load shedding.
Remember one sentence: do as little work as possible on the request path, because at the edge every microsecond is multiplied by the entire system's throughput.
---