Definition
- Horizontal scaling – run many identical stateless instances behind a load balancer, adding more freely.
- Auto-scaling – HPA and KEDA automatically add or remove instances based on CPU, request rate, or queue depth.
- Caching – serve hot reads from Redis so the database is not hammered during spikes.
- Async processing – push slow work onto a Kafka queue so requests return fast and the queue absorbs spikes.
- Database scaling – connection pooling, read replicas, and sharding prevent the database from becoming the bottleneck.
- Rate limiting and load shedding – the gateway throttles or rejects the least important traffic to protect the core path.
- Monitoring – watch latency percentiles, error rates, queue depth, and saturation to detect and respond early.
Imagine the National Bank of India launches a popular new fixed deposit, and suddenly a huge crowd floods the branch. A well-run bank handles the rush by opening more counters (horizontal scaling), calling in staff automatically as the queue grows (auto-scaling), posting today's interest rates on a board at the door so people do not each ask a clerk (caching), handing out token numbers so customers do not stand blocking the counter while paperwork is processed in the back room (async processing), and making sure the central records office can handle many clerks reading at once (database scaling). A supervisor watches the queue and politely asks non-urgent visitors to come back later (load shedding).
Understanding the Concepts
What is horizontal scaling and why must the service be stateless?
Horizontal scaling is the foundation for handling traffic surges. It works only if the service is stateless, meaning no per-user session is stored in local memory. Any instance can handle any request, which means you can run ten or a hundred identical instances behind a load balancer. Session or user state goes into a shared store like Redis.
package com.bank.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.session.data.redis.config.annotation.web.http.EnableRedisHttpSession;
@Configuration
@EnableRedisHttpSession(maxInactiveIntervalInSeconds = 1800)
public class SessionConfig {
// Sessions stored in Redis, not in JVM heap.
// Any instance can serve any request.
}
Stateless service behind load balancer:
Request 1 -> Instance A (reads session from Redis)
Request 2 -> Instance C (reads same session from Redis)
Request 3 -> Instance B (reads same session from Redis)
All three see the same session data.
How does auto-scaling work?
Auto-scaling makes instance addition automatic. The Horizontal Pod Autoscaler (HPA) watches a signal like CPU utilization or requests per second and increases the replica count when the signal crosses a target. For bursty, queue-driven workloads, KEDA scales on queue depth, which reacts faster than CPU for spiky loads.
# Kubernetes HPA configuration
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: account-service-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: account-service
minReplicas: 3
maxReplicas: 30
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "100"
Normal traffic: 3 pods, CPU at 30%
Salary-day surge: CPU hits 70% -> HPA scales to 15 pods in 2 minutes
Surge passes: CPU drops -> HPA scales back to 3 pods over 10 minutes
Why is caching the single biggest lever?
Caching removes load from the database, which is usually the first tier to buckle. With a cache-aside strategy the service checks Redis first, and only on a miss does it hit the database and then populate the cache with a short time-to-live (TTL). A thousand balance checks per second mostly hit Redis in microseconds instead of hammering PostgreSQL.
package com.bank.service;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.cache.annotation.CacheEvict;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class AccountService {
private final AccountRepository repository;
public AccountService(AccountRepository repository) {
this.repository = repository;
}
@Cacheable(value = "balances", key = "#accountId")
public BigDecimal getBalance(String accountId) {
// Cache miss: reads from PostgreSQL
return repository.findBalanceByAccountId(accountId);
}
@CacheEvict(value = "balances", key = "#accountId")
public void updateBalance(String accountId, BigDecimal newBalance) {
repository.updateBalance(accountId, newBalance);
// Cache evicted: next read will refresh from DB
}
}
# Redis cache configuration
spring:
cache:
type: redis
redis:
time-to-live: 30s
cache-null-values: false
Without cache: 1000 req/s * DB query = 1000 DB queries/s
With cache: 1000 req/s * 95% hit rate = 50 DB queries/s
Result: 20x reduction in database load during surge
How does asynchronous processing keep latency low?
Async processing moves slow work off the request path. Instead of doing everything synchronously while the customer waits, the service does the minimal critical step and drops a message onto Kafka for the rest. A transfer updates balances synchronously but publishes an event so that statement generation, notifications, and analytics happen later.
package com.bank.transfer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class TransferService {
private final AccountRepository accountRepo;
private final KafkaTemplate<String, TransferEvent> kafka;
public TransferService(AccountRepository accountRepo,
KafkaTemplate<String, TransferEvent> kafka) {
this.accountRepo = accountRepo;
this.kafka = kafka;
}
@Transactional
public TransferResult transfer(String from, String to,
BigDecimal amount) {
// Critical path: update balances synchronously
accountRepo.debit(from, amount);
accountRepo.credit(to, amount);
// Non-critical path: publish event for async consumers
kafka.send("transfer-events",
new TransferEvent(from, to, amount));
return TransferResult.success("Transfer completed");
}
}
Sync-only flow: debit + credit + statement + notification + analytics = 800ms
Async flow: debit + credit = 50ms (rest happens via Kafka consumers later)
Result: 16x latency improvement on the request path
How do you scale the database?
The database is usually the first tier to buckle under surge. The main techniques are connection pooling, read replicas, and sharding.
# HikariCP connection pool configuration
spring:
datasource:
hikari:
maximum-pool-size: 20
minimum-idle: 5
connection-timeout: 2000
idle-timeout: 300000
max-lifetime: 1800000
package com.bank.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;
import javax.sql.DataSource;
import java.util.Map;
@Configuration
public class ReadReplicaConfig {
@Bean
public DataSource routingDataSource(DataSource primary,
DataSource replica) {
AbstractRoutingDataSource routing = new AbstractRoutingDataSource() {
@Override
protected Object determineCurrentLookupKey() {
return TransactionSynchronizationManager
.isCurrentTransactionReadOnly()
? "replica" : "primary";
}
};
routing.setTargetDataSources(Map.of(
"primary", primary,
"replica", replica
));
routing.setDefaultTargetDataSource(primary);
return routing;
}
}
Database scaling techniques:
Connection pool: prevents opening a new connection per request
Read replicas: route heavy reads to copies, writes to primary
Sharding: split data across DBs by account_id for write scale
What is rate limiting and load shedding?
Rate limiting and load shedding protect the system when it is genuinely beyond capacity. The gateway rejects or throttles the least important traffic (for example, defer analytics calls) to protect the core path like transfers and balance checks.
package com.bank.gateway;
import org.springframework.cloud.gateway.filter.ratelimit.KeyResolver;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import reactor.core.publisher.Mono;
@Configuration
public class RateLimitConfig {
@Bean
public KeyResolver userKeyResolver() {
return exchange -> Mono.just(
exchange.getRequest().getHeaders()
.getFirst("X-Client-Id")
);
}
}
# Gateway rate limiting with Redis
spring:
cloud:
gateway:
routes:
- id: account-service
uri: lb://account-service
filters:
- name: RequestRateLimiter
args:
redis-rate-limiter.replenishRate: 100
redis-rate-limiter.burstCapacity: 200
redis-rate-limiter.requestedTokens: 1
key-resolver: "#{@userKeyResolver}"
Rate limit response:
HTTP 429 Too Many Requests
Retry-After: 5
X-RateLimit-Remaining: 0
How do you monitor and respond to a surge?
Watch latency percentiles, error rates, queue depth, and saturation. Set up dashboards and alerts so the team can respond before the system breaks.
package com.bank.monitoring;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Component;
@Component
public class TransferMetrics {
private final Timer transferTimer;
private final Counter transferCounter;
private final Counter failureCounter;
public TransferMetrics(MeterRegistry registry) {
this.transferTimer = Timer.builder("transfer.duration")
.description("Transfer processing time")
.register(registry);
this.transferCounter = Counter.builder("transfer.total")
.description("Total transfers")
.register(registry);
this.failureCounter = Counter.builder("transfer.failures")
.description("Failed transfers")
.register(registry);
}
public Timer getTransferTimer() { return transferTimer; }
public Counter getTransferCounter() { return transferCounter; }
public Counter getFailureCounter() { return failureCounter; }
}
Key Prometheus alerts:
transfer_duration_seconds{quantile="0.99"} > 2 -> P99 latency alert
transfer_failures_total rate > 0.05 -> Error rate alert
hikaricp_connections_active == hikaricp_connections_max -> Pool exhaustion
kafka_consumer_lag > 10000 -> Consumer falling behind
What is the step-by-step principle for handling surges?
The guiding principle is: start simple, measure, find the current bottleneck, relieve exactly that tier, and repeat. Never scale everything speculatively.
| Stage | Bottleneck | Fix |
|---|---|---|
| 1 | Slow DB queries | Add indexes, optimize queries |
| 2 | Single app instance | Stateless app + load balancer |
| 3 | DB read load | Cache (Redis) + read replicas + CDN |
| 4 | Sync slow work | Async queue (Kafka) + rate limiting |
| 5 | Single DB write | Shard by account_id |
| 6 | Manual scaling | Auto-scaling (HPA/KEDA) + multi-region |
Visual Overview
Traffic Surge
│
▼
┌──────────────────────────────┐
│ CDN (static assets) │
└──────────┬───────────────────┘
│
▼
┌──────────────────────────────┐
│ API Gateway │
│ Rate Limiting + Load Shed │
└──────────┬───────────────────┘
│
┌───────┴───────┐
▼ ▼
┌───────┐ ┌───────┐ ┌───────┐
│Pod 1 │ │Pod 2 │ │Pod N │ <- Auto-scaled
│(stateless) │(stateless) │(stateless)
└───┬───┘ └───┬───┘ └───┬───┘
│ │ │
▼ ▼ ▼
┌──────────────────────────────┐
│ Redis Cache │ <- 95% reads served here
└──────────┬───────────────────┘
│ cache miss
▼
┌──────────────────────────────┐
│ PostgreSQL (Primary + Replicas) │
└──────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Kafka (async processing) │
│ Statements, Notifications │
└──────────────────────────────┘
Key Takeaway
Keeping an application available during a huge traffic surge is about keeping request handling cheap and fast and never letting any single tier become the bottleneck. The foundation is horizontal scaling, which only works if the service is stateless, so sessions go into Redis and any instance can handle any request. Auto-scaling adds instances automatically as CPU or request rate rises, and KEDA reacts to queue depth for bursty workloads.
The single biggest lever for a read-heavy surge is caching, because it removes load from the database. In the account service at National Bank of India, the vast majority of traffic is balance checks, which are reads of slowly changing data. With cache-aside in Redis with a thirty-second TTL, what would have been tens of thousands of database reads per second became a trickle, keeping the connection pool from exhausting and write transactions flowing. Async processing keeps latency low by moving slow work off the request path: a transfer updates balances synchronously and returns immediately, then publishes an event so statement generation and notifications happen later via Kafka consumers.
Database scaling backs all of this up with a tuned connection pool, read replicas for heavy read traffic, and sharding for truly massive write volume. Rate limiting and load shedding at the gateway protect the core path when the system is genuinely beyond capacity, deferring analytics and non-critical calls. Monitoring with Prometheus alerts on P99 latency, error rates, pool exhaustion, and consumer lag gives the team visibility to respond before things break.
Remember one sentence: keep the app stateless, cache hot reads, offload slow work to a queue, scale the database with replicas and pooling, and always measure to find the one bottleneck before scaling.
---