Definition
- Round-robin – Distribute incoming requests evenly across backends by cycling in order: server 0, 1, 2, 0, 1, 2.
- AtomicInteger – A thread-safe integer using hardware compare-and-swap;
getAndIncrement()gives each caller a unique tick. - Math.floorMod – Always returns a non-negative result, unlike
%which can return negative after integer overflow. - Health-aware skip – Skip unhealthy backends and advance to the next healthy one.
- Weighted round-robin – Bigger instances get proportionally more traffic by appearing multiple times in the server list.
National Bank of India has five teller counters. A round-robin host at the door sends the first customer to counter 1, the next to counter 2, and so on, looping back after counter 5. Every counter gets an equal share. The host keeps a single tally counter, and because many customers arrive at once, the tally must be updated atomically so two customers are never sent to the same counter by mistake.
Understanding the Concepts
What is round-robin load balancing?
Given a list of N backends, request number k goes to backend k % N. The only shared state is the counter k.
package com.bank.lb;
public class SimpleRoundRobin {
public static void main(String[] args) {
String[] servers = {"payment-1", "payment-2", "payment-3"};
for (int request = 0; request < 9; request++) {
int index = request % servers.length;
System.out.println("Request " + request + " -> " + servers[index]);
}
}
}
Expected output:
Request 0 -> payment-1
Request 1 -> payment-2
Request 2 -> payment-3
Request 3 -> payment-1
Request 4 -> payment-2
Request 5 -> payment-3
Request 6 -> payment-1
Request 7 -> payment-2
Request 8 -> payment-3
Why is AtomicInteger necessary for thread safety?
If you use a plain int and do counter++, that is not atomic. Two threads can read the same value and route to the same backend.
package com.bank.lb;
public class ThreadSafetyProblem {
// WRONG: plain int
// int counter = 0;
// int index = counter++; // NOT ATOMIC
// Thread A reads 5, Thread B reads 5
// Both pick server 5 % 3 = 2
// Server distribution skews
// CORRECT: AtomicInteger
// AtomicInteger counter = new AtomicInteger();
// int index = counter.getAndIncrement(); // ATOMIC CAS
// Thread A gets 5, Thread B gets 6
// Different servers picked
public static void main(String[] args) {
System.out.println("WRONG: plain int counter++ is not atomic");
System.out.println(" Two threads read same value -> same server -> skew");
System.out.println();
System.out.println("CORRECT: AtomicInteger.getAndIncrement() is atomic CAS");
System.out.println(" Each thread gets a distinct, advancing value");
}
}
Expected output:
WRONG: plain int counter++ is not atomic
Two threads read same value -> same server -> skew
CORRECT: AtomicInteger.getAndIncrement() is atomic CAS
Each thread gets a distinct, advancing value
How do you implement a thread-safe round-robin?
Use AtomicInteger.getAndIncrement() with Math.floorMod for the index.
package com.bank.lb;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
public class RoundRobinLoadBalancer {
private final List<String> servers;
private final AtomicInteger counter = new AtomicInteger(0);
public RoundRobinLoadBalancer(List<String> servers) {
if (servers == null || servers.isEmpty()) {
throw new IllegalArgumentException("Server list must not be empty");
}
this.servers = List.copyOf(servers);
}
public String nextServer() {
int tick = counter.getAndIncrement();
int index = Math.floorMod(tick, servers.size());
return servers.get(index);
}
public static void main(String[] args) {
var lb = new RoundRobinLoadBalancer(
List.of("payment-1:8080", "payment-2:8080", "payment-3:8080"));
for (int i = 0; i < 6; i++) {
System.out.println("Request " + i + " -> " + lb.nextServer());
}
}
}
Expected output:
Request 0 -> payment-1:8080
Request 1 -> payment-2:8080
Request 2 -> payment-3:8080
Request 3 -> payment-1:8080
Request 4 -> payment-2:8080
Request 5 -> payment-3:8080
Why use Math.floorMod instead of %?
When the counter overflows past Integer.MAX_VALUE it wraps to negative. The % operator returns a negative index, causing IndexOutOfBoundsException. floorMod always returns non-negative.
package com.bank.lb;
public class FloorModVsPercent {
public static void main(String[] args) {
int negative = Integer.MIN_VALUE; // -2147483648
int n = 3;
System.out.println("% operator: " + negative + " % " + n + " = " + (negative % n));
System.out.println("floorMod: " + negative + " floorMod " + n + " = "
+ Math.floorMod(negative, n));
System.out.println();
System.out.println("% gives -2 -> list.get(-2) throws IndexOutOfBoundsException!");
System.out.println("floorMod gives 1 -> valid index, routing continues correctly");
}
}
Expected output:
% operator: -2147483648 % 3 = -2
floorMod: -2147483648 floorMod 3 = 1
% gives -2 -> list.get(-2) throws IndexOutOfBoundsException!
floorMod gives 1 -> valid index, routing continues correctly
How do you add health-aware skipping?
Track each backend's health and skip unhealthy ones.
package com.bank.lb;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
public class HealthAwareRoundRobin {
private final List<ServerInfo> servers;
private final AtomicInteger counter = new AtomicInteger(0);
public HealthAwareRoundRobin(List<ServerInfo> servers) {
this.servers = servers;
}
public ServerInfo nextHealthyServer() {
int startTick = counter.getAndIncrement();
for (int attempt = 0; attempt < servers.size(); attempt++) {
int index = Math.floorMod(startTick + attempt, servers.size());
ServerInfo server = servers.get(index);
if (server.isHealthy()) {
return server;
}
}
throw new IllegalStateException("No healthy servers available");
}
public record ServerInfo(String address, boolean isHealthy) {
public boolean isHealthy() { return isHealthy; }
}
public static void main(String[] args) {
var lb = new HealthAwareRoundRobin(List.of(
new ServerInfo("payment-1:8080", true),
new ServerInfo("payment-2:8080", false), // DOWN
new ServerInfo("payment-3:8080", true)
));
for (int i = 0; i < 4; i++) {
System.out.println("Request " + i + " -> " + lb.nextHealthyServer().address());
}
}
}
Expected output:
Request 0 -> payment-1:8080
Request 1 -> payment-3:8080 (skipped unhealthy payment-2)
Request 2 -> payment-1:8080
Request 3 -> payment-3:8080
How does weighted round-robin work?
Bigger instances get more traffic. Expand the list so a weight-2 server appears twice.
package com.bank.lb;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
public class WeightedRoundRobin {
private final List<String> expandedList;
private final AtomicInteger counter = new AtomicInteger(0);
public WeightedRoundRobin(List<WeightedServer> servers) {
this.expandedList = new ArrayList<>();
for (WeightedServer ws : servers) {
for (int i = 0; i < ws.weight(); i++) {
expandedList.add(ws.address());
}
}
}
public String nextServer() {
int index = Math.floorMod(counter.getAndIncrement(), expandedList.size());
return expandedList.get(index);
}
public record WeightedServer(String address, int weight) {}
public static void main(String[] args) {
var lb = new WeightedRoundRobin(List.of(
new WeightedServer("big-server:8080", 3),
new WeightedServer("small-server:8080", 1)
));
for (int i = 0; i < 8; i++) {
System.out.println("Request " + i + " -> " + lb.nextServer());
}
}
}
Expected output:
Request 0 -> big-server:8080
Request 1 -> big-server:8080
Request 2 -> big-server:8080
Request 3 -> small-server:8080
Request 4 -> big-server:8080
Request 5 -> big-server:8080
Request 6 -> big-server:8080
Request 7 -> small-server:8080
How does round-robin compare to other strategies?
| Strategy | Pros | Cons | Best for |
|---|---|---|---|
| Round-robin | Simple, even distribution, lock-free | Ignores actual load per request | Equal-cost requests |
| Least-connections | Adapts to real load | More shared state needed | Variable-cost requests |
| Random | Stateless, simple | Less even for small N | Large server pools |
| Consistent hashing | Sticky to key, minimal remapping | Uneven with few nodes | Caching, session affinity |
package com.bank.lb;
public class StrategyComparison {
public static void main(String[] args) {
System.out.println("Round-robin: treats every request as one equal tick");
System.out.println(" -> Good when requests are similar cost");
System.out.println(" -> Bad when one request takes 5 seconds, another takes 5ms");
System.out.println();
System.out.println("Least-connections: routes to server with fewest active requests");
System.out.println(" -> Adapts to variable request cost");
System.out.println(" -> Needs shared state of active connection counts");
}
}
Expected output:
Round-robin: treats every request as one equal tick
-> Good when requests are similar cost
-> Bad when one request takes 5 seconds, another takes 5ms
Least-connections: routes to server with fewest active requests
-> Adapts to variable request cost
-> Needs shared state of active connection counts
What about contention at extreme throughput?
At very high throughput even a single AtomicInteger becomes a cache-line contention hotspot. Shard the counter for that case.
package com.bank.lb;
public class ContentionAtScale {
public static void main(String[] args) {
System.out.println("Single AtomicInteger: fine for 99% of services");
System.out.println(" CAS on one cache line -> threads compete");
System.out.println();
System.out.println("Extreme throughput (millions req/sec):");
System.out.println(" Option 1: Shard the counter (per-thread or per-core)");
System.out.println(" Option 2: LongAdder-style striped counter");
System.out.println(" Trade: perfect ordering for throughput");
System.out.println(" Round-robin only needs roughly even, not globally perfect");
}
}
Expected output:
Single AtomicInteger: fine for 99% of services
CAS on one cache line -> threads compete
Extreme throughput (millions req/sec):
Option 1: Shard the counter (per-thread or per-core)
Option 2: LongAdder-style striped counter
Trade: perfect ordering for throughput
Round-robin only needs roughly even, not globally perfect
Visual Overview
Round-Robin Load Balancer:
Requests: R0 R1 R2 R3 R4 R5 R6 R7 R8
│ │ │ │ │ │ │ │ │
AtomicInt: 0 1 2 3 4 5 6 7 8
floorMod: 0 1 2 0 1 2 0 1 2
│ │ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
Servers: [0] [1] [2] [0] [1] [2] [0] [1] [2]
Health-Aware:
Servers: [payment-1: UP] [payment-2: DOWN] [payment-3: UP]
R0 -> index 0 -> payment-1 (UP) -> route
R1 -> index 1 -> payment-2 (DOWN) -> skip
index 2 -> payment-3 (UP) -> route
Weighted (big:3, small:1):
Expanded: [big] [big] [big] [small]
R0->big R1->big R2->big R3->small R4->big ...
Key Takeaway
A round-robin load balancer distributes requests evenly across backends by cycling in order: request k goes to backend k modulo N. The only shared state is the counter, and that is where thread safety matters. A plain int with counter++ is not atomic: two threads can read the same value and route to the same backend. AtomicInteger's getAndIncrement() is a single indivisible compare-and-swap that gives each caller a distinct tick.
The floorMod detail matters: over billions of requests the int counter overflows past Integer.MAX_VALUE and wraps negative. The % operator would produce a negative index causing IndexOutOfBoundsException, while Math.floorMod always returns a non-negative result. A production balancer also skips unhealthy nodes and supports weighted round-robin for unequal instances.
At National Bank of India, Pankaj implemented this for the payment service load balancer. One instance got overloaded while others sat idle because the original code used a non-atomic plain int counter. Switching to AtomicInteger with floorMod fixed the skew, health-aware skipping stopped routing to a crashed instance, and weighted expansion gave the larger instances proportionally more traffic.
Remember one sentence: implement round-robin with AtomicInteger.getAndIncrement() mapped to a backend via Math.floorMod for overflow safety, extend it with health-aware skipping and weighted expansion, and the result is a lock-free, thread-safe, resilient load balancer.
---