Definition
- Connection pooling – keeps established TCP+TLS connections open and reuses them across requests, targeting over 99% reuse.
- HTTP/2 multiplexing – many concurrent requests share a single connection, reducing the number of connections needed.
- Hybrid rate limiting – a local in-memory token bucket checked instantly per request, synced with Redis every ~100ms for global consistency.
- Async batched logging – log events enqueued in memory and flushed by a background worker, removing I/O from the request path.
- Request coalescing – identical concurrent requests share one downstream call; also called single-flight or request collapsing.
Imagine the National Bank of India dispatch desk sending documents to the central vault. If a courier sets up a new secure route for each document, that is wasteful; instead the courier keeps routes open and reuses them (connection pooling). If the clerk phones head office to check each customer's limit, that is slow; instead he keeps a local tally and syncs every few seconds (hybrid rate limiting). If he files paperwork after every customer, the line jams; instead he drops notes in a tray for batch filing (async logging). If three customers ask the same question at once, he answers once and tells all three (coalescing).
Understanding the Concepts
How does connection pooling cut overhead?
Opening a new connection to an upstream service means a TCP handshake plus a TLS handshake, costing several round trips and CPU. A connection pool keeps connections alive and reuses them. HTTP/2 multiplexes many requests over one connection.
package com.bank.gateway;
import io.netty.channel.ChannelOption;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import reactor.netty.http.client.HttpClient;
import reactor.netty.resources.ConnectionProvider;
import java.time.Duration;
@Configuration
public class ConnectionPoolConfig {
@Bean
public HttpClient httpClient() {
ConnectionProvider provider = ConnectionProvider.builder("gateway-pool")
.maxConnections(500) // Max connections
.maxIdleTime(Duration.ofSeconds(20)) // Keep-alive
.maxLifeTime(Duration.ofMinutes(5)) // Recycle connections
.pendingAcquireTimeout(Duration.ofMillis(500))
.build();
return HttpClient.create(provider)
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000)
.responseTimeout(Duration.ofMillis(3000));
}
}
// Without pool: every request = new TCP + TLS handshake = ~5ms overhead
// With pool: >99% of requests reuse existing connections = ~0.05ms
// Savings: 100x reduction in connection setup cost
// HTTP/2 multiplexing: 100 concurrent requests share 1 connection
// Instead of 100 connections, you need ~1-5 connections per backend
How does hybrid rate limiting work?
A naive distributed limiter does a Redis round trip per request. The hybrid approach keeps a local token bucket and syncs with Redis periodically.
package com.bank.gateway.ratelimit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class HybridRateLimiter {
private final AtomicInteger localTokens;
private final int maxTokens;
private final RedisRateLimitSync redisSync;
public HybridRateLimiter(int maxTokens, RedisRateLimitSync redisSync) {
this.maxTokens = maxTokens;
this.localTokens = new AtomicInteger(maxTokens);
this.redisSync = redisSync;
// Sync with Redis every 100ms
Executors.newSingleThreadScheduledExecutor()
.scheduleAtFixedRate(this::syncWithRedis, 100, 100,
TimeUnit.MILLISECONDS);
}
public boolean tryAcquire(String clientId) {
// Local check: instant, no network
int remaining = localTokens.decrementAndGet();
return remaining >= 0;
}
private void syncWithRedis() {
// Periodic sync: report local usage, get global remaining
int consumed = maxTokens - localTokens.get();
int globalRemaining = redisSync.syncAndGetRemaining(consumed);
localTokens.set(Math.min(globalRemaining, maxTokens));
}
}
// Naive: 50,000 req/s * 0.5ms Redis call = 25,000ms Redis I/O per second
// Hybrid: 10 syncs/s * 0.5ms = 5ms Redis I/O per second
// Savings: 5,000x reduction in Redis calls
// Trade-off: global count is ~100ms stale between syncs (acceptable)
How does async batched logging work?
Synchronous log writes make the request wait for disk or network I/O. Async logging enqueues events and flushes in batches.
package com.bank.gateway.logging;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.ArrayList;
import java.util.List;
public class AsyncAccessLogger {
private final LinkedBlockingQueue<AccessLogEntry> queue =
new LinkedBlockingQueue<>(10_000);
public void log(AccessLogEntry entry) {
// Non-blocking enqueue: ~0.001ms
queue.offer(entry);
}
// Background worker flushes every 100ms or when batch is full
public void flush() {
List<AccessLogEntry> batch = new ArrayList<>();
queue.drainTo(batch, 1000);
if (!batch.isEmpty()) {
writeBatchToDisk(batch); // One disk write for 1000 entries
}
}
private void writeBatchToDisk(List<AccessLogEntry> batch) {
// Write all entries in one I/O operation
}
}
// Synchronous: 50,000 req/s * 0.3ms per disk write = 15,000ms I/O // Async batch: 50 flushes/s * 1ms per batch write = 50ms I/O // Per-request cost: 0.3ms → 0.001ms (300x faster) // Caution: crash loses buffered entries; for critical audit, use durable queue
How does request coalescing work?
When many identical requests are in flight simultaneously, coalescing makes them share one downstream call instead of each making its own.
package com.bank.gateway;
import reactor.core.publisher.Mono;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
public class RequestCoalescer {
private final ConcurrentMap<String, Mono<Object>> inFlight =
new ConcurrentHashMap<>();
public Mono<Object> coalesce(String cacheKey,
java.util.function.Supplier<Mono<Object>> fetcher) {
return inFlight.computeIfAbsent(cacheKey, key -> {
// First request: create the actual downstream call
return fetcher.get()
.doFinally(signal -> inFlight.remove(key))
.cache(); // Share the result with all waiters
});
}
}
// 100 clients request GET /api/v1/rates/fd simultaneously:
// Without coalescing: 100 calls to rates-service
// With coalescing: 1 call to rates-service, result shared with 100 clients
// IMPORTANT: only for identical, idempotent reads!
// NEVER coalesce: POST /api/v1/transfers (each transfer is unique)
How do these techniques interact?
All four techniques reduce per-request cost independently and stack together.
// Combined savings per request: // Connection pool: 5ms → 0.05ms (saved 4.95ms) // Hybrid rate limit: 0.5ms → 0.01ms (saved 0.49ms) // Async logging: 0.3ms → 0.001ms (saved 0.299ms) // Request coalescing: N calls → 1 call (saved (N-1) * latency) // Total per-request cost: ~2ms → ~0.06ms // Gateway throughput: 20K req/s → 300K+ req/s per replica
What are common mistakes? (wrong vs correct)
// WRONG: connection pool too small → contention
// 10 connections for 50,000 req/s → massive queuing
// CORRECT: size pool based on throughput and upstream capacity
// 500 connections with keep-alive → >99% reuse
// WRONG: coalescing POST /transfers (unique operations!)
// Two different transfers merged → one customer's money moves, other's doesn't
// CORRECT: coalesce only identical, idempotent GETs
// GET /rates/fd → same for everyone → safe to coalesce
// WRONG: unbounded log queue → OOM on spike
// CORRECT: bounded queue with drop policy for overflow
What should a senior engineer know?
A senior engineer measures where per-request time actually goes, applies these techniques in order of impact, and monitors the metrics to validate improvements. They never coalesce state-changing operations.
// Senior optimization checklist:
// 1. Measure: profile per-request time breakdown
// 2. Connection pool: configure size, keep-alive, monitor reuse rate
// 3. Hybrid rate limit: tune sync interval vs precision trade-off
// 4. Async logging: bounded queue, batch size, flush interval
// 5. Coalescing: only identical idempotent reads
// 6. Monitor: connection reuse %, Redis call rate, log flush latency
// 7. The goal: remove any fixed per-request cost at the edge
What are the edge cases?
Connection pooling can backfire if the upstream is a serverless function that resets between invocations, leaving stale connections in the pool. Coalescing must never apply to authenticated or personalized requests because two requests from different users would share results. Hybrid rate limiting drifts by at most one sync interval, which is acceptable for API throttling but not for monetary limits.
package com.bank.gateway;
public class EdgeCases {
public static void main(String[] args) {
// Edge case 1: serverless upstream resets connections
// Pool returns a stale connection -> 502 Bad Gateway
// Fix: enable health-check pings on idle connections
// Edge case 2: coalescing authenticated requests
// WRONG: coalesce GET /api/accounts/me (different per user!)
// CORRECT: coalesce GET /api/rates/fd (same for everyone)
// Edge case 3: rate-limit drift
// Local bucket at 100ms sync: user may get 100ms worth of extra tokens
// Acceptable for API throttling, NOT for per-second money limits
System.out.println("Edge case 1: stale pooled connections to serverless");
System.out.println("Edge case 2: never coalesce user-specific responses");
System.out.println("Edge case 3: sync drift acceptable for throttling, not money");
}
}
Expected output:
Edge case 1: stale pooled connections to serverless
Edge case 2: never coalesce user-specific responses
Edge case 3: sync drift acceptable for throttling, not money
Visual Overview
Per-Request Overhead Reduction
──────────────────────────────
Without optimization:
Request → new connection (5ms) → rate limit Redis (0.5ms)
→ sync log (0.3ms) → downstream call → response
Total overhead: ~5.8ms per request
With optimization:
Request → pooled connection (0.05ms) → local rate limit (0.01ms)
→ async log (0.001ms) → downstream call → response
Total overhead: ~0.06ms per request
Request Coalescing:
100 identical requests ──▶ Coalescer ──▶ 1 downstream call
◀──────────── result shared with all 100
Hybrid Rate Limiting:
Request → local bucket (instant) → allowed/denied
│
└──every 100ms──▶ Redis (sync global state)
Key Takeaway
Cutting per-request overhead means eliminating the repeated, wasteful work the gateway does for each request, because at the edge every fixed cost is multiplied by the entire system's throughput. The four techniques are connection pooling (reuse TCP+TLS connections, targeting over 99% reuse), hybrid rate limiting (local token bucket synced with Redis every 100ms instead of a Redis call per request), async batched logging (enqueue and flush in the background, not inline), and request coalescing (identical concurrent requests share one downstream call).
At National Bank of India, applying all four cut the per-request overhead from about 5.8ms to 0.06ms. Connection pooling alone removed the biggest cost because each new TCP+TLS handshake was 5ms that multiplied by 50,000 requests per second was consuming enormous CPU and network resources. Hybrid rate limiting dropped Redis calls from 50,000 per second to 10, trading a few hundred milliseconds of global staleness for a massive reduction in latency and Redis load.
The nuances are safety and tuning. Coalescing applies only to identical, idempotent reads, never to distinct money-moving operations, because merging two different transfers would be a correctness disaster. The connection pool must be sized to avoid both contention (too small) and upstream exhaustion (too large). The rate-limit sync interval balances latency against how strict the limit must be.
Remember one sentence: remove any fixed per-request cost at the edge through connection pooling, hybrid rate limiting, async logging, and request coalescing, because at the edge every microsecond is paid by the whole system.
---