Back to Scaling, Load Balancing & Performance
Q3Question 3 of 20

Explain the Load Balancer and the Rate Limiter

Key Concepts
  • A load balancer distributes requests across instances answering 'where does this go,' while a rate limiter caps requests per client answering 'should this be allowed right now.'
  • Load balancer algorithms: round-robin, least-connections, weighted, IP-hash, consistent hashing.
  • Rate limiter algorithms: fixed-window, sliding-window, token bucket, leaky bucket.
  • Token bucket permits controlled bursts up to bucket capacity while enforcing a steady long-run average rate.
  • A distributed rate limiter must share counters in Redis so the limit holds across the whole fleet.
  • The rate limiter runs at the edge or gateway, rejecting excess before backends consume capacity.
  • They are used together, not interchangeably: rate limiter at the edge, load balancer behind it.

Definition

A load balancer spreads requests across instances to answer 'where does this go,' while a rate limiter caps requests per client over time to answer 'should this be allowed right now' : distribution versus protection.
  • Load balancer – distributes incoming requests across multiple backend instances so no single instance is overwhelmed, using algorithms like round-robin and least-connections.
  • Rate limiter – restricts how many requests a client may make in a time window, protecting the system from overload and abuse, using algorithms like token bucket and sliding-window.
  • Key distinction – the load balancer decides placement among healthy instances; the rate limiter decides admission based on client quota.
  • Deployment – the rate limiter runs at the edge or gateway, the load balancer distributes admitted requests behind it.
  • Distributed state – a rate limiter across many instances must share counters in Redis, otherwise N instances each allow the full limit.

Imagine the National Bank of India's branch at rush hour. The load balancer is the receptionist who looks at the open counters and sends each customer to whichever one is free, so the work is spread evenly. The rate limiter is the security guard at the door who makes sure no single person submits two hundred forms in one minute, because that would jam the whole branch. One directs traffic to the right place, the other controls how much traffic is allowed in the first place. Together they keep the branch fast and fair.

Understanding the Concepts

What is a load balancer?

A load balancer sits in front of a pool of identical backend instances and selects one for each request. Its purpose is to spread work evenly so throughput is maximized and no instance saturates while others idle.

JAVA
package com.bank.loadbalancer;

import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

public class SimpleRoundRobin {

    private final List<String> backends;
    private final AtomicInteger index = new AtomicInteger(0);

    public SimpleRoundRobin(List<String> backends) {
        this.backends = backends;
    }

    public String route() {
        return backends.get(index.getAndIncrement() % backends.size());
    }

    public static void main(String[] args) {
        var lb = new SimpleRoundRobin(
            List.of("payment-1:8080", "payment-2:8080", "payment-3:8080"));
        for (int i = 0; i < 6; i++) {
            System.out.println("Request " + (i + 1) + " -> " + lb.route());
        }
    }
}
TEXT
Expected output:
Request 1 -> payment-1:8080
Request 2 -> payment-2:8080
Request 3 -> payment-3:8080
Request 4 -> payment-1:8080
Request 5 -> payment-2:8080
Request 6 -> payment-3:8080

What are the load balancer algorithms?

Each algorithm suits a different workload profile.

| Algorithm | How It Works | Best For |
|---|---|---|
| Round-robin | Rotate through instances | Uniform cheap requests |
| Least-connections | Pick instance with fewest active requests | Variable request costs |
| Weighted round-robin | Bias toward higher-capacity instances | Mixed-capacity fleet |
| IP-hash | Map client IP to one instance | Session stickiness |
| Consistent hashing | Map key to instance on a hash ring | Distributed caches |

What is a rate limiter?

A rate limiter restricts how many requests a client may make within a time window. When exceeded, the server responds with HTTP 429 Too Many Requests.

JAVA
package com.bank.ratelimit;

import java.math.BigDecimal;
import java.util.concurrent.atomic.AtomicLong;
import java.util.logging.Logger;

public class SimpleTokenBucket {

    private static final Logger LOG = Logger.getLogger(SimpleTokenBucket.class.getName());
    private final long capacity;
    private final long refillRate; // tokens per second
    private AtomicLong tokens;
    private long lastRefillTimestamp;

    public SimpleTokenBucket(long capacity, long refillRate) {
        this.capacity = capacity;
        this.refillRate = refillRate;
        this.tokens = new AtomicLong(capacity);
        this.lastRefillTimestamp = System.nanoTime();
    }

    public synchronized boolean tryConsume() {
        refill();
        if (tokens.get() > 0) {
            tokens.decrementAndGet();
            return true;
        }
        return false; // 429 Too Many Requests
    }

    private void refill() {
        long now = System.nanoTime();
        long elapsed = (now - lastRefillTimestamp) / 1_000_000_000;
        long newTokens = elapsed * refillRate;
        tokens.set(Math.min(capacity, tokens.get() + newTokens));
        lastRefillTimestamp = now;
    }

    public static void main(String[] args) {
        var bucket = new SimpleTokenBucket(5, 2); // 5 capacity, 2/sec refill
        for (int i = 0; i < 8; i++) {
            boolean allowed = bucket.tryConsume();
            LOG.info("Request " + (i + 1) + ": " + (allowed ? "ALLOWED" : "REJECTED (429)"));
        }
    }
}
TEXT
Expected output:
Request 1: ALLOWED
Request 2: ALLOWED
Request 3: ALLOWED
Request 4: ALLOWED
Request 5: ALLOWED
Request 6: REJECTED (429)
Request 7: REJECTED (429)
Request 8: REJECTED (429)

What are the rate limiter algorithms?

| Algorithm | Behavior | Trade-off |
|---|---|---|
| Fixed-window | Count per clock window | Simple but double-burst at boundary |
| Sliding-window log | Timestamps of recent requests | Accurate but memory-heavy |
| Sliding-window counter | Blend current and previous window | Cheap approximation, smooths boundary |
| Token bucket | Refill tokens at steady rate | Permits controlled bursts |
| Leaky bucket | Constant outflow rate | Smooths bursty input into steady stream |

How is a load balancer different from a rate limiter? (comparison)

These two components solve different problems and are used together, not interchangeably.

| Aspect | Load Balancer | Rate Limiter |
|---|---|---|
| Question answered | Where does this request go? | Should this request be allowed? |
| Operates on | Instance pool | Client identity (user, API key) |
| Goal | Spread work evenly | Protect from overload and abuse |
| Response on limit | Route to another instance | HTTP 429 Too Many Requests |
| Deployment | Behind the edge | At the edge or gateway |
| Algorithm examples | Round-robin, least-connections | Token bucket, sliding-window |

How do you implement a distributed rate limiter?

A rate limiter across many instances must share counters in Redis. If each of five instances keeps a local counter, each allows the full limit and the effective limit is five times too high.

JAVA
package com.bank.ratelimit;

import io.github.bucket4j.Bandwidth;
import io.github.bucket4j.Bucket;
import io.github.bucket4j.BucketConfiguration;
import io.github.bucket4j.distributed.proxy.ProxyManager;
import java.time.Duration;
import java.util.function.Supplier;

public class DistributedRateLimiter {

    private final ProxyManager<String> proxyManager; // backed by Redis

    public DistributedRateLimiter(ProxyManager<String> proxyManager) {
        this.proxyManager = proxyManager;
    }

    public boolean isAllowed(String userId) {
        Supplier<BucketConfiguration> config = () -> BucketConfiguration.builder()
            .addLimit(Bandwidth.simple(100, Duration.ofMinutes(1))) // 100 req/min
            .build();

        Bucket bucket = proxyManager.builder()
            .build("rate:" + userId, config);

        return bucket.tryConsume(1);
        // All 5 gateway instances share this Redis-backed bucket.
        // Total limit: 100/min per user across the entire fleet.
    }
}
TEXT
WRONG: Per-instance counters
  5 instances x 100 limit = 500 effective limit per user

CORRECT: Shared Redis counter
  5 instances sharing 1 Redis bucket = 100 effective limit per user

Where do you place them in the request path?

The rate limiter runs at the gateway and rejects excess before backends consume capacity. The load balancer distributes admitted requests across instances.

DIAGRAM
  Client
    │
    ▼
  ┌──────────────────────────────┐
  │  API Gateway                  │
  │  1. Rate Limiter (per user)   │  <-- reject if 429
  │  2. Auth check                │
  │  3. Route to service          │
  └──────────────┬───────────────┘
                 │
  ┌──────────────┴───────────────┐
  │  Load Balancer               │  <-- pick healthy instance
  │  (least-connections)         │
  └───┬──────────┬───────────────┘
      ▼          ▼
  ┌────────┐ ┌────────┐
  │ Pod 1  │ │ Pod 2  │
  └────────┘ └────────┘

What are common mistakes?

The most common mistake is keeping rate-limiter state local to each instance.

JAVA
package com.bank.antipattern;

// WRONG: Local counter per instance. 5 instances = 5x the intended limit.
public class WrongRateLimiter {
    private int count = 0; // per-instance
    public boolean allow() { return ++count <= 100; }
}

// CORRECT: Shared counter in Redis. Limit holds across the whole fleet.
// Use Bucket4j backed by Redis via Redisson or Spring Cloud Gateway
// RedisRateLimiter with key-resolver by user ID.

What tools are commonly used?

For load balancing: Nginx, HAProxy, Envoy, AWS ALB, Azure Application Gateway, Kubernetes Service with kube-proxy, and service mesh (Istio). For rate limiting: Spring Cloud Gateway RedisRateLimiter, Kong, Bucket4j, Resilience4j RateLimiter, and Redis for shared state.

Visual Overview

DIAGRAM
  Request flow showing both components:

  Client Request
      │
      ▼
  ┌─────────────────────────┐
  │     RATE LIMITER         │
  │  "Should I allow this?"  │
  │  Token Bucket in Redis   │
  │  Per-user, per-endpoint  │
  │                          │
  │  Over limit? -> 429      │
  └───────────┬──────────────┘
              │ allowed
              ▼
  ┌─────────────────────────┐
  │     LOAD BALANCER        │
  │  "Where does this go?"   │
  │  Least-connections       │
  │  Health-checked pool     │
  └───┬─────────┬────────────┘
      │         │
      ▼         ▼
  ┌────────┐ ┌────────┐
  │ Pod 1  │ │ Pod 2  │
  └────────┘ └────────┘

Key Takeaway

A load balancer and a rate limiter are two complementary traffic-management components that solve different problems. The load balancer answers 'where should this request go' by distributing requests across healthy instances using algorithms like round-robin or least-connections. The rate limiter answers 'should this request be allowed at all right now' by capping requests per client using algorithms like token bucket or sliding-window, returning HTTP 429 when exceeded.

In the request path at National Bank of India, the rate limiter runs at the API gateway and rejects excess before it consumes any backend capacity. The load balancer distributes the admitted requests across instances. During peak hours, one instance was overwhelmed while others idled, which the load balancer fixed with least-connections. Separately, a buggy mobile client in a retry loop fired thousands of requests a minute, which the rate limiter fixed by capping that client and returning 429.

The critical distributed-systems subtlety is that a rate limiter across many instances must share its counters in Redis. If each of five instances keeps a local counter, each allows the full limit and the effective limit is five times too high.

Remember one sentence: the load balancer is about distribution across healthy instances, the rate limiter is about protection through admission control, put the rate limiter at the edge and the load balancer behind it, and share the rate-limiter state in Redis so the limit holds globally.

---

Share
Use ← → arrow keys to navigate