Definition
- Shared Database Bottleneck – finite connection limit and I/O; more app instances open more connections and can exhaust the limit faster.
- Connection Proxy (PgBouncer) – multiplexes a bounded number of real DB connections regardless of how many instances exist.
- Third-Party API Quota – a fixed external rate limit shared by all instances; more instances race to consume it.
- Distributed Rate Limiter – enforces per-tenant sub-quotas centrally (e.g., Redis-backed) against a shared external budget.
- Hot Kafka Partition – all events for one customer key land on one partition and one consumer; adding consumers does nothing.
- Composite Partition Key – spreading a large customer's events across partitions (e.g., customerId + accountId) while preserving finer-grained ordering.
National Bank of India can hire as many tellers as it likes, but if every transaction must be stamped by the one manager who holds the only key to the vault, then hiring a hundred more tellers just means a hundred more people queued at the manager's door. If a giant corporate client sends ten thousand transactions, the manager is swamped and Rajesh's simple withdrawal waits. More tellers is the wrong lever when the manager is the chokepoint.
Understanding the Concepts
Why is "just add more instances" insufficient?
Horizontal scaling works when the constraint is CPU or memory in the stateless application layer. But when the bottleneck is a shared downstream resource, adding instances adds more clients competing for the same scarce thing.
package com.bank.multitenant;
public class WrongLever {
public static void main(String[] args) {
System.out.println("Stateless bottleneck: CPU at 95% → add instances → CORRECT");
System.out.println();
System.out.println("Shared DB bottleneck: DB connections at 100% → add instances");
System.out.println(" → more instances open MORE connections → exhausts limit FASTER");
System.out.println(" → WRONG lever, makes it WORSE");
}
}
Stateless bottleneck: CPU at 95% → add instances → CORRECT Shared DB bottleneck: DB connections at 100% → add instances → more instances open MORE connections → exhausts limit FASTER → WRONG lever, makes it WORSE
How do you fix a shared database bottleneck?
A connection proxy like PgBouncer multiplexes a bounded number of real connections. Per-tenant query concurrency limits prevent one tenant from monopolizing them.
package com.bank.multitenant;
public class DbBottleneckFix {
public static void main(String[] args) {
int appInstances = 100;
int connectionsPerInstance = 10;
int totalWithoutProxy = appInstances * connectionsPerInstance;
int proxyLimit = 50;
System.out.println("Without proxy: " + totalWithoutProxy + " connections (exceeds DB limit)");
System.out.println("With PgBouncer proxy: " + proxyLimit + " real connections (bounded)");
System.out.println();
System.out.println("Plus per-tenant query concurrency:");
System.out.println(" BigCorp: max 5 concurrent queries");
System.out.println(" Cannot hold all 50 connections");
}
}
Without proxy: 1000 connections (exceeds DB limit)
With PgBouncer proxy: 50 real connections (bounded)
Plus per-tenant query concurrency:
BigCorp: max 5 concurrent queries
Cannot hold all 50 connections
How do you fix a third-party API quota bottleneck?
A shared distributed rate limiter enforces per-tenant sub-quotas so each tenant gets a fair slice of the total external budget.
package com.bank.multitenant;
public class ExternalQuotaFix {
public static void main(String[] args) {
int totalQuota = 1000; // calls per second from payment gateway
int tenants = 100;
int perTenantQuota = totalQuota / tenants;
System.out.println("External API quota: " + totalQuota + "/sec (shared)");
System.out.println("Per-tenant sub-quota: " + perTenantQuota + "/sec");
System.out.println();
System.out.println("BigCorp cannot consume more than " + perTenantQuota + "/sec");
System.out.println("Remaining " + (totalQuota - perTenantQuota)
+ "/sec available for all other tenants");
}
}
External API quota: 1000/sec (shared)
Per-tenant sub-quota: 10/sec
BigCorp cannot consume more than 10/sec
Remaining 990/sec available for all other tenants
How do you fix a hot Kafka partition?
Use a composite partition key that spreads a large customer's events across many partitions while preserving finer-grained ordering.
package com.bank.multitenant;
public class HotPartitionFix {
public static void main(String[] args) {
// WRONG: key = customerId → all BigCorp events on ONE partition
String wrongKey = "bigcorp";
System.out.println("Wrong key: '" + wrongKey + "' → all events on partition "
+ Math.abs(wrongKey.hashCode() % 10));
// CORRECT: composite key spreads across partitions
String[] accounts = {"bigcorp:ACC001", "bigcorp:ACC002", "bigcorp:ACC003"};
for (String key : accounts) {
System.out.println("Composite key: '" + key + "' → partition "
+ Math.abs(key.hashCode() % 10));
}
System.out.println("BigCorp's accounts now span multiple partitions and consumers");
}
}
Wrong key: 'bigcorp' → all events on partition 2 Composite key: 'bigcorp:ACC001' → partition 7 Composite key: 'bigcorp:ACC002' → partition 4 Composite key: 'bigcorp:ACC003' → partition 1 BigCorp's accounts now span multiple partitions and consumers
How do you diagnose the real bottleneck?
The telltale sign is low instance CPU and memory but high latency and errors. The instances are waiting on something shared downstream.
package com.bank.multitenant;
public class BottleneckDiagnosis {
public static void main(String[] args) {
System.out.println("Instance CPU: 15% (LOW)");
System.out.println("Instance memory: 40% (LOW)");
System.out.println("P99 latency: 5000ms (HIGH)");
System.out.println("Error rate: 25% (HIGH)");
System.out.println();
System.out.println("Diagnosis: instances are idle, waiting on shared downstream resource");
System.out.println("Adding instances will NOT help (wrong lever)");
System.out.println("Fix: identify and bound the shared resource");
}
}
Instance CPU: 15% (LOW)
Instance memory: 40% (LOW)
P99 latency: 5000ms (HIGH)
Error rate: 25% (HIGH)
Diagnosis: instances are idle, waiting on shared downstream resource
Adding instances will NOT help (wrong lever)
Fix: identify and bound the shared resource
How do you compare the three bottlenecks?
| Bottleneck | Why More Instances Fails | Fix |
|---|---|---|
| Shared database | More connections exhaust DB limit | PgBouncer + per-tenant query concurrency + sharding |
| Third-party API quota | More instances race for fixed quota | Distributed per-tenant sub-quotas |
| Hot Kafka partition | One partition = one consumer, extras idle | Composite key to spread across partitions |
What are common mistakes?
Reflexively adding instances without diagnosing which resource is actually saturated.
package com.bank.multitenant;
// WRONG: "Latency is high? Scale out!"
// Adds 10 instances → 10 more DB connections → DB connection limit hit → WORSE
// CORRECT: "Instance CPU is low but latency is high → bottleneck is downstream"
// Fix the downstream resource: proxy, quota, partition key
What are common interview follow-up questions?
- How do you implement a distributed per-tenant rate limiter with Redis?
- What happens when a Kafka partition rebalance occurs after adding partitions?
- How do you decide when to shard the database versus adding read replicas?
- What metrics tell you the difference between a stateless and a shared-resource bottleneck?
Visual Overview
Three Shared Bottlenecks
────────────────────────
1. Shared Database:
App (100 instances) ──► PgBouncer (50 conns) ──► PostgreSQL
│ per-tenant concurrency limit
└ BigCorp: max 5 queries at once
2. Third-Party API:
App (100 instances) ──► Distributed Limiter ──► Payment Gateway
│ 1000/sec total (fixed quota)
└ BigCorp: 10/sec sub-quota
3. Hot Kafka Partition:
Before: key=bigcorp → all on partition 2 → one consumer
After: key=bigcorp:ACC001 → spread across partitions 1,4,7
→ multiple consumers
Key Takeaway
A system can be fully horizontally scaled at the stateless layer and still be brought to its knees by one large customer, because the real bottleneck is often a shared downstream resource. Adding instances is the wrong lever when the bottleneck is a shared database, a third-party API quota, or a hot Kafka partition. The telltale sign is low instance CPU but high latency: the instances are idle, waiting on something shared.
For a shared database, use a connection proxy and per-tenant query concurrency. For a third-party API quota, use a distributed per-tenant sub-quota. For a hot Kafka partition, use a composite partition key that spreads load across multiple partitions. Each fix acts at the point of contention.
Diagnose first, then act. The fix for a stateless bottleneck (add instances) is the opposite of the fix for a shared-resource bottleneck (bound and partition the shared resource). Fairness and isolation must live where the contention actually is.
Remember one sentence: horizontal scaling of the stateless tier fixes one kind of bottleneck; for shared databases, external quotas, and hot partitions, enforce fairness at the point of contention.
---