Definition
- Principle – never scale speculatively; let measurement, not guesswork, drive each step.
- Stage 1 – optimize queries and indexes on a single database for the cheapest, highest-impact fix.
- Stage 2 – make the app stateless and add a load balancer so instances can be added freely.
- Stage 3 – add caching (Redis), read replicas, and a CDN to remove read load from the database.
- Stage 4 – introduce async message queues (Kafka) and rate limiting to smooth write spikes.
- Stage 5 – split into microservices and shard the database by account ID for independent scaling.
- Stage 6 – add auto-scaling (HPA/KEDA) and multi-region deployment for geographic resilience.
Imagine the National Bank of India opens as a single tiny branch with one clerk. As the town grows, you do not immediately build a fifty-story headquarters. Instead you watch where the queue actually forms. First the clerk is slow finding files, so you organize the filing cabinet better (indexes). Then one clerk cannot keep up, so you add more clerks and a receptionist (load balancer). Then people keep asking the same question, so you put a notice board at the door (caching). Then paperwork piles up, so you add a token system and a back office (async queues). Then the single branch is too big, so you split into departments (microservices). Finally you open branches in other cities (multi-region).
Understanding the Concepts
Why start simple and measure?
Premature scaling wastes effort and adds complexity you may never need, while under-scaling causes outages. Every optimization just moves the bottleneck somewhere else, so there is no point scaling a tier that is not the current constraint. You measure, relieve the actual constraint, and the bottleneck shifts to the next tier.
package com.bank.monitoring;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Component;
@Component
public class BottleneckDetector {
private final Timer dbQueryTimer;
private final Timer cacheTimer;
private final Timer externalCallTimer;
public BottleneckDetector(MeterRegistry registry) {
this.dbQueryTimer = Timer.builder("bottleneck.db.query")
.description("Database query latency")
.register(registry);
this.cacheTimer = Timer.builder("bottleneck.cache.lookup")
.description("Cache lookup latency")
.register(registry);
this.externalCallTimer = Timer.builder("bottleneck.external.call")
.description("External service call latency")
.register(registry);
}
}
The iterative loop:
1. Measure under real load
2. Find the single current bottleneck
3. Relieve exactly that bottleneck
4. The bottleneck shifts to the next tier
5. Repeat
Stage 1: How do indexes and query optimization help?
At one user to a few thousand, the simplest design is a single application server talking to a single database. The first bottleneck is almost always slow database queries. Adding the right indexes turns a balance lookup from a full table scan into an instant index seek.
-- BEFORE: full table scan (slow as data grows)
SELECT balance FROM accounts WHERE account_id = 'SAV2001';
-- Execution plan: Seq Scan on accounts cost=0.00..1250.00
-- FIX: add an index
CREATE INDEX idx_accounts_account_id ON accounts(account_id);
-- AFTER: index seek (fast regardless of table size)
SELECT balance FROM accounts WHERE account_id = 'SAV2001';
-- Execution plan: Index Scan using idx_accounts_account_id cost=0.00..8.27
package com.bank.repository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.CrudRepository;
import java.math.BigDecimal;
public interface AccountRepository extends CrudRepository<Account, Long> {
// WRONG: N+1 query pattern
// List<Account> findAll(); // then loop and call getTransactions()
// CORRECT: fetch join to avoid N+1
@Query("SELECT a FROM Account a LEFT JOIN FETCH a.recentTransactions " +
"WHERE a.accountId = :accountId")
Account findWithTransactions(String accountId);
}
Impact of indexing:
Before: balance lookup 120ms (full scan of 1M rows)
After: balance lookup 2ms (index seek)
Cost: zero infrastructure change, just one SQL statement
Stage 2: How do you make the app stateless?
As traffic grows, the single application server becomes the bottleneck. Make the app stateless by moving session state out of instance memory into a shared store, then run several instances behind a load balancer.
package com.bank.config;
import org.springframework.session.data.redis.config.annotation.web.http.EnableRedisHttpSession;
import org.springframework.context.annotation.Configuration;
@Configuration
@EnableRedisHttpSession
public class StatelessConfig {
// Sessions stored in Redis, not in JVM heap
// Any instance can handle any request
}
# Nginx load balancer config
upstream account-service {
server 192.168.1.10:8080;
server 192.168.1.11:8080;
server 192.168.1.12:8080;
}
server {
listen 80;
location /api/ {
proxy_pass http://account-service;
}
}
Before: 1 instance handling 500 req/s (maxed out)
After: 3 instances behind LB -> 1500 req/s capacity
Stage 3: How do caching and read replicas reduce database load?
The database becomes the bottleneck again under read-heavy load. Three fixes apply together: caching hot data in Redis, adding read replicas, and putting a CDN in front of static assets.
package com.bank.service;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class BalanceService {
private final AccountRepository repository;
public BalanceService(AccountRepository repository) {
this.repository = repository;
}
@Cacheable(value = "balances", key = "#accountId")
public BigDecimal getBalance(String accountId) {
return repository.findBalanceByAccountId(accountId);
}
}
Read scaling strategy:
Cache (Redis): 95% of reads served from memory (microseconds)
Read replicas: 5% cache misses go to replicas, not primary
CDN: static assets served from edge (zero server load)
Primary DB: handles only writes and rare cache misses
Stage 4: How do async queues and rate limiting smooth spikes?
Write throughput and slow synchronous chains become the bottleneck. A service does its critical step and publishes an event to Kafka, letting downstream work happen later and letting the queue absorb spikes.
package com.bank.transfer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class AsyncTransferService {
private final AccountRepository accountRepo;
private final KafkaTemplate<String, TransferEvent> kafka;
public AsyncTransferService(AccountRepository accountRepo,
KafkaTemplate<String, TransferEvent> kafka) {
this.accountRepo = accountRepo;
this.kafka = kafka;
}
@Transactional
public String transfer(String from, String to, BigDecimal amount) {
accountRepo.debit(from, amount);
accountRepo.credit(to, amount);
// Async: statement, notification, analytics happen later
kafka.send("transfer-events",
new TransferEvent(from, to, amount));
return "Transfer completed";
}
}
Queue as shock absorber:
Spike: 10,000 transfers/s burst
Producer: writes to Kafka at burst rate
Consumer: drains at steady 2,000/s
Queue depth: absorbs the 8,000/s difference
Result: no dropped requests, statements catch up in minutes
Stage 5: How do microservices and sharding provide independent scaling?
As the codebase and team grow, the monolith becomes the bottleneck. Split into microservices so each part scales, deploys, and fails independently. Shard the database by account ID so write capacity grows beyond a single primary.
Monolith -> Microservices:
account-service (PostgreSQL, sharded by account_id)
transfer-service (PostgreSQL, references account-service API)
statement-service (Cassandra, high write volume)
notification-service (Redis, ephemeral)
package com.bank.shard;
public class ShardRouter {
private static final int SHARD_COUNT = 4;
public int getShard(String accountId) {
// Consistent hash: same account always goes to same shard
return Math.abs(accountId.hashCode()) % SHARD_COUNT;
}
}
Sharding by account_id:
Shard 0: accounts A000-A249 (DB node 1)
Shard 1: accounts A250-A499 (DB node 2)
Shard 2: accounts A500-A749 (DB node 3)
Shard 3: accounts A750-A999 (DB node 4)
Write capacity: 4x single primary
Stage 6: How do auto-scaling and multi-region complete the picture?
At the million-user scale, manual scaling and single-region deployment become the bottleneck. Auto-scaling matches capacity to demand automatically, and multi-region puts the system in multiple geographic regions for lower latency and resilience.
# Kubernetes HPA for auto-scaling
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: transfer-service-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: transfer-service
minReplicas: 3
maxReplicas: 50
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
Multi-region deployment:
Region 1 (Mumbai): Primary for Indian customers
Region 2 (Singapore): Replica for Southeast Asia
Global LB: Routes to nearest healthy region
Failover: If Mumbai is down, Singapore serves all traffic
What is the complete evolution summary?
| Stage | Users | Bottleneck | Fix | Cost |
|---|---|---|---|---|
| 1 | 1-1K | Slow queries | Indexes | Zero |
| 2 | 1K-10K | Single server | Stateless + LB | Low |
| 3 | 10K-100K | DB reads | Cache + replicas + CDN | Medium |
| 4 | 100K-500K | Sync chains | Async queues + rate limit | Medium |
| 5 | 500K-1M | Monolith | Microservices + sharding | High |
| 6 | 1M+ | Manual ops | Auto-scale + multi-region | High |
Visual Overview
Stage 1: Single Server
┌────────┐ ┌──────┐
│ App │────>│ DB │ <- Optimize queries & indexes
└────────┘ └──────┘
Stage 2: Stateless + Load Balancer
┌──────┐ ┌────────┐
│ LB │────>│ App x3 │────>┌──────┐
└──────┘ └────────┘ │ DB │
└──────┘
Stage 3: Cache + Read Replicas
┌──────┐ ┌────────┐ ┌───────┐ ┌──────────┐
│ CDN │ │ LB │────>│App x3 │────>│ Redis │
└──────┘ └────────┘ └───────┘ └────┬─────┘
│ miss │
▼ ▼
┌──────────────────────┐
│ Primary + 2 Replicas │
└──────────────────────┘
Stage 4-6: Full Architecture
┌──────┐ ┌──────┐ ┌──────────────┐ ┌───────┐
│ CDN │ │ GW │──>│ Microservices│──>│ Redis │
└──────┘ │ Rate │ │ (auto-scaled)│ └───┬───┘
│ Limit│ └──────┬───────┘ │
└──────┘ │ ┌───▼────┐
▼ │Sharded │
┌─────────┐ │ DBs │
│ Kafka │ └────────┘
└─────────┘
Key Takeaway
Evolving an architecture from one user to over a million is an iterative loop: start simple, measure under real load to find the single current bottleneck, relieve exactly that, and repeat. The reason for that discipline is that premature scaling wastes effort and adds complexity you may never need, while every optimization just moves the bottleneck to the next tier.
At the beginning with a handful of users, a single application server and database is right. The first bottleneck is almost always slow queries, and adding an index on account_id turned a balance lookup at National Bank of India from a full scan into an instant seek. As traffic grows, the app becomes the bottleneck, so you make it stateless and add a load balancer. Then the database buckles under reads, so you add Redis caching, read replicas, and a CDN. Then write spikes and sync chains slow things down, so you introduce Kafka and rate limiting. Then the monolith is too big, so you split into microservices and shard. Finally, manual scaling and single-region are the constraint, so you add auto-scaling and multi-region.
The senior-level framing is that you never jump straight to the end state. You arrive at it one measured step at a time, and at each step the question is always the same: what is the single thing limiting us right now, and what is the cheapest way to relieve it?
Remember one sentence: start simple, measure, find the bottleneck, scale that one thing, and repeat, because the right architecture is always the simplest one that handles the current load.
---