Definition
- Load balancer – sits in front of N identical instances and selects one for each request using a configured algorithm.
- Round-robin – rotates through instances in order; fair for uniform requests but ignores actual instance load.
- Least-connections – sends to the instance with the fewest active requests; adapts to variable request costs.
- Consistent hashing – routes by a key so the same client or account sticks to the same instance, helping caches.
- Health checks – periodically probe each instance and remove unhealthy ones from the rotation.
- Kubernetes Service – provides a stable virtual IP and DNS name, tracking healthy Pod endpoints via readiness probes.
Imagine the National Bank of India has several identical counters open and a crowd of customers arriving. The receptionist at the door decides who goes to which counter. The simplest fair rule is rotation: one each in turn (round-robin). A smarter rule is to look at which counter has the shortest queue (least-connections). For customers needing ongoing paperwork, the same customer always goes to the same clerk (consistent hashing). The receptionist only directs people to counters that are actually open (health checks), never to a closed one.
Understanding the Concepts
What is a load balancer?
A load balancer sits in front of N identical instances and, for each incoming request, selects one instance according to a configured algorithm. The goal is to spread work evenly so no instance is saturated while others are idle.
package com.bank.loadbalancer;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
/**
* Simple round-robin load balancer for National Bank of India.
* In production, use Kubernetes Service or Nginx/Envoy.
*/
public class RoundRobinBalancer {
private final List<String> instances;
private final AtomicInteger counter = new AtomicInteger(0);
public RoundRobinBalancer(List<String> instances) {
this.instances = instances;
}
public String nextInstance() {
int index = counter.getAndIncrement() % instances.size();
return instances.get(index);
}
public static void main(String[] args) {
var balancer = new RoundRobinBalancer(
List.of("payment-1:8080", "payment-2:8080", "payment-3:8080"));
for (int i = 0; i < 6; i++) {
System.out.println("Request " + (i + 1) + " -> " + balancer.nextInstance());
}
}
}
Expected output:
Request 1 -> payment-1:8080
Request 2 -> payment-2:8080
Request 3 -> payment-3:8080
Request 4 -> payment-1:8080
Request 5 -> payment-2:8080
Request 6 -> payment-3:8080
How does round-robin work?
Round-robin cycles through instances in order, giving each the next request in turn. It is fair when requests are uniform and instances are identical. Its weakness is that it ignores how busy each instance actually is.
Round-robin with 3 instances:
Request 1 -> Instance A
Request 2 -> Instance B
Request 3 -> Instance C
Request 4 -> Instance A (cycle repeats)
Problem: if Request 1 is a slow fraud check (5 seconds),
Instance A is still busy but round-robin sends Request 4 to it anyway.
How does least-connections work?
Least-connections sends each new request to the instance currently handling the fewest active requests. This adapts to uneven request costs.
package com.bank.loadbalancer;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class LeastConnectionsBalancer {
private final List<String> instances;
private final Map<String, Integer> activeConnections = new ConcurrentHashMap<>();
public LeastConnectionsBalancer(List<String> instances) {
this.instances = instances;
instances.forEach(i -> activeConnections.put(i, 0));
}
public synchronized String nextInstance() {
return instances.stream()
.min((a, b) -> activeConnections.get(a) - activeConnections.get(b))
.orElseThrow();
}
public void onRequestStart(String instance) {
activeConnections.merge(instance, 1, Integer::sum);
}
public void onRequestEnd(String instance) {
activeConnections.merge(instance, -1, Integer::sum);
}
}
Expected behavior:
Instance A: 5 active connections (slow fraud checks)
Instance B: 1 active connection
Instance C: 2 active connections
Next request -> Instance B (fewest active)
How does consistent hashing work?
Consistent hashing maps a key (like account ID) to an instance on a hash ring. The same key always hits the same instance, which helps local caching.
package com.bank.loadbalancer;
import java.util.SortedMap;
import java.util.TreeMap;
public class ConsistentHashBalancer {
private final SortedMap<Integer, String> ring = new TreeMap<>();
public void addInstance(String instance, int virtualNodes) {
for (int i = 0; i < virtualNodes; i++) {
int hash = (instance + "-" + i).hashCode();
ring.put(hash, instance);
}
}
public String getInstanceFor(String accountId) {
int hash = accountId.hashCode();
SortedMap<Integer, String> tail = ring.tailMap(hash);
int key = tail.isEmpty() ? ring.firstKey() : tail.firstKey();
return ring.get(key);
}
public static void main(String[] args) {
var balancer = new ConsistentHashBalancer();
balancer.addInstance("payment-1", 100);
balancer.addInstance("payment-2", 100);
balancer.addInstance("payment-3", 100);
System.out.println("SAV2001 -> " + balancer.getInstanceFor("SAV2001"));
System.out.println("SAV2002 -> " + balancer.getInstanceFor("SAV2002"));
System.out.println("SAV2001 -> " + balancer.getInstanceFor("SAV2001")); // same
}
}
Expected output:
SAV2001 -> payment-2
SAV2002 -> payment-1
SAV2001 -> payment-2 (same instance every time for same key)
How do health checks work?
Health checks periodically probe each instance and remove any that fail from the rotation. Without them, traffic would blindly include dead instances.
# Kubernetes readiness probe for payment-service
apiVersion: v1
kind: Pod
spec:
containers:
- name: payment-service
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
Expected behavior:
Pod starts -> readiness probe fails for 10s -> NOT in Service endpoints
Readiness probe passes -> added to Service endpoints -> receives traffic
Pod crashes -> readiness fails 3 times -> removed from endpoints -> no traffic
How does a Kubernetes Service distribute requests?
A Kubernetes Service provides a stable virtual IP and DNS name, and kube-proxy spreads requests across the healthy Pod endpoints.
apiVersion: v1
kind: Service
metadata:
name: payment-service
spec:
selector:
app: payment-service
ports:
- port: 80
targetPort: 8080
type: ClusterIP
Expected behavior:
DNS: payment-service.default.svc.cluster.local -> 10.96.0.100 (virtual IP)
kube-proxy distributes to healthy pods:
-> 10.244.0.5:8080 (pod 1)
-> 10.244.0.6:8080 (pod 2)
-> 10.244.0.7:8080 (pod 3)
Round-robin-style at the connection level.
How do load balancing algorithms compare? (comparison table)
| Algorithm | Strategy | Best For | Weakness |
|---|---|---|---|
| Round-robin | Rotate in order | Uniform cheap requests | Ignores instance load |
| Least-connections | Fewest active requests | Variable request costs | Slightly more overhead |
| Weighted round-robin | Bias to capacity | Mixed-capacity fleet | Manual weight tuning |
| Consistent hashing | Key-based affinity | Caching, sticky sessions | Uneven if keys skewed |
| IP-hash | Client IP affinity | Session stickiness | Uneven with NAT |
What should a senior engineer know?
kube-proxy balances TCP connections, not individual HTTP requests. For long-lived or HTTP/2 multiplexed connections, you need a layer-seven balancer or service mesh to truly balance request load.
Layer 4 (kube-proxy): balances TCP connections
Problem: one HTTP/2 connection carries 1000 requests to one pod
Other pods sit idle.
Layer 7 (Envoy mesh): balances individual HTTP requests
Fix: each request is independently routed to the least-busy pod.
Visual Overview
┌──────────────────────────────────────────┐
│ CLIENT REQUESTS │
└────────────────────┬─────────────────────┘
│
┌────────────────────┴─────────────────────┐
│ LOAD BALANCER │
│ Algorithm: round-robin / least-conn │
│ Health checks: remove unhealthy pods │
└───┬──────────┬──────────┬────────────────┘
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ Pod 1 │ │ Pod 2 │ │ Pod 3 │
│payment │ │payment │ │payment │
│:8080 │ │:8080 │ │:8080 │
└────────┘ └────────┘ └────────┘
Kubernetes Service:
payment-service (ClusterIP 10.96.0.100)
-> tracks healthy pod endpoints
-> kube-proxy distributes connections
Key Takeaway
Distributing requests equally across N instances is the job of a load balancer, and the key is choosing the algorithm that matches your request-cost profile. Round-robin cycles through instances in order and is perfectly fair for uniform requests. Least-connections sends each request to the instance with the fewest active requests, adapting naturally to variable costs. Consistent hashing maps a key to an instance on a ring so the same account always hits the same instance, which helps local caching.
Health checks are what make all of this safe. The load balancer periodically probes each instance and removes any that fail from the rotation. In Kubernetes, a Service gives a stable virtual IP and DNS name and tracks healthy Pod endpoints through readiness probes, giving round-robin-style balancing with no application code.
The nuance I always mention is that kube-proxy balances TCP connections, not individual HTTP requests. For long-lived or HTTP/2 connections, you need a layer-seven ingress controller or a service mesh to truly balance request load.
Remember one sentence: put a load balancer in front of N stateless instances, gate the rotation with health checks, and pick the algorithm to fit your workload: round-robin for uniform requests, least-connections for variable ones, and consistent hashing when affinity matters.
---