Scaling, Load Balancing & Performance

HPA adds more pods, VPA makes each pod bigger, Cluster Autoscaler adds more nodes, and KEDA scales on event signals like queue depth instead of CPU: different layers for different problems.

A load balancer spreads requests across N healthy instances using an algorithm like round-robin or least-connections, health checks keep dead instances out of the rotation, and a Kubernetes Service does all of this automatically behind a stable address.

A load balancer spreads requests across instances to answer 'where does this go,' while a rate limiter caps requests per client over time to answer 'should this be allowed right now' : distribution versus protection.

Azure layers its load balancers by scope and OSI layer: Load Balancer is regional L4, Application Gateway is regional L7 with a firewall, Front Door is global L7 with edge routing and failover, and AKS auto-provisions an Azure Load Balancer for a LoadBalancer-type Service.

Distributed per-user rate limiting keeps one shared, atomic counter per user in Redis so the quota is enforced across every instance, usually at the gateway, rejecting excess with 429.

Back-of-envelope estimation turns daily active users into QPS, bytes per day into bytes per year, rate times payload into bandwidth, and the hot fraction into cache size, all anchored by the latency numbers every engineer should know.

Little's Law gives required concurrency as throughput times latency, and you divide by what one instance can handle to get the instance count, sizing CPU, memory, and the connection pool to match.

A hot row serializes writers on one lock, so either shard the counter into N sub-rows that writers spread across and sum on read, or buffer increments in Redis and flush the aggregate to the DB periodically.

Fanout-on-write precomputes timelines for O(1) reads but amplifies writes by follower count, fanout-on-read keeps writes cheap but makes reads expensive, and the hybrid uses fanout-on-write for normal users and fanout-on-read for celebrities to avoid writing to millions of timelines per post.

Round-robin cycles backends in order; an AtomicInteger's getAndIncrement modulo the server count gives a lock-free, thread-safe next pick.

Do not poll the whole database every second for due jobs; store jobs ordered by execute-at in a Redis sorted set so finding due jobs is one range query, and bucket by time so each tick inspects only the relevant slice.

Scaling microservices means scaling each service independently: stateless replicas behind load balancers, an API gateway in front, containers auto-scaled by Kubernetes, the database scaled with caching and replicas, slow work offloaded asynchronously, all guided by monitoring.

Reaching a million requests per second means specifying partitioning, request routing, replication, capacity planning, hot-key handling, and node-failure behaviour, because adding servers alone does nothing until data is partitioned and requests are routed to the node that owns each key.

Sharding permanently changes how the application is written, so first find the actual saturated resource and exhaust every cheaper, reversible fix before you shard.

The Dockerfile builds one immutable image, docker-compose runs a small set of containers together for local development, and Kubernetes runs them in production with self-healing, rolling updates, and automatic horizontal scaling.

Autoscaling the application tier against a saturated downstream adds more callers to the queue at the real bottleneck, and the idle-waiting replicas report low CPU that tricks the autoscaler into scaling further.

For the last unit, use in-JVM locks within one process, database locking across instances sharing a DB, a Redis SETNX lock across instances without a shared row, or a single-consumer queue that serializes requests so no lock is needed.

Never run one load balancer; run a redundant pair sharing a floating virtual IP with heartbeat failover, add health checks, and use DNS or multi-zone failover so the loss of any single balancer never takes the system down.

Vertical scaling makes one instance bigger and hits a ceiling with downtime, while horizontal scaling adds more instances behind a load balancer for near-unlimited capacity and high availability.

Multi-level rate limiting works because per-user, per-IP, per-provider, and global each catch a distinct pattern, and a token bucket per provider enables clean failover instead of hammering an exhausted provider.