Definition
- Multi-level rate limiting – Separate limits at per-user, per-IP, per-provider, and global scopes, evaluated together.
- Per-user – Stops one account monopolizing capacity (e.g., 5 OTPs/min per user).
- Per-IP – Catches one source attacking many accounts (e.g., credential stuffing from one IP).
- Per-provider – Keeps outbound calls within a downstream's contractual rate.
- Global – Protects total system capacity from any flood larger than you can serve.
- Token bucket per provider – Tracks each downstream's remaining allowance so you fail over to one with tokens.
Imagine National Bank of India controlling a crowd at a service hall. One rule cannot stop a greedy customer, spot a troublemaker using many names, and respect each printer's speed limit. So the bank uses several rules: no single customer gets too many, no single entrance lets too many through, each printer is asked to print only as fast as it can, and the whole hall has a cap. When one printer's quota is used up, the clerk sends the next job to a printer that still has quota.
Understanding the Concepts
Why is a single limiter insufficient?
A single limiter at any scope has blind spots that other scopes cover.
package com.bank.ratelimit;
public class SingleLimiterBlindSpots {
public static void main(String[] args) {
System.out.println("Single GLOBAL limiter blind spots:");
System.out.println(" Cannot tell one abusive user from 1000 legitimate ones");
System.out.println(" Throttling everyone to stop one abuser punishes good users");
System.out.println();
System.out.println("Single PER-USER limiter blind spots:");
System.out.println(" Attacker spreads across 10,000 accounts -> each under per-user limit");
System.out.println(" Aggregate is massive but per-user looks normal");
System.out.println();
System.out.println("Single PER-IP limiter blind spots:");
System.out.println(" Distributed attack from 10,000 IPs -> each IP under limit");
System.out.println(" Corporate NAT: 500 legitimate users share one IP -> all throttled");
System.out.println();
System.out.println("Solution: COMPOSE all levels so each covers the other's blind spot");
}
}
Expected output:
Single GLOBAL limiter: blind to who is causing the load
Single PER-USER limiter: blind to distributed attack across accounts
Single PER-IP limiter: blind to distributed attack + hurts shared IPs
Solution: COMPOSE all levels
How do the levels work together?
A request must pass ALL applicable levels. Whichever pattern is present, the matching level catches it.
package com.bank.ratelimit;
public class MultiLevelRateLimiting {
public static void main(String[] args) {
System.out.println("Multi-level rate limiting:");
System.out.println(" Level 1 - Per-user: max 5 OTPs/min per account");
System.out.println(" Level 2 - Per-IP: max 20 requests/min per IP");
System.out.println(" Level 3 - Per-provider: max 100 SMS/sec to Twilio");
System.out.println(" Level 4 - Global: max 10,000 requests/sec total");
System.out.println();
System.out.println(" Request must pass ALL levels to proceed.");
System.out.println(" Greedy user? Level 1 catches it.");
System.out.println(" Script from one IP? Level 2 catches it.");
System.out.println(" Provider saturated? Level 3 catches it.");
System.out.println(" Massive flood? Level 4 catches it.");
}
}
Expected output:
Multi-level rate limiting:
Level 1 - Per-user: max 5 OTPs/min per account
Level 2 - Per-IP: max 20 requests/min per IP
Level 3 - Per-provider: max 100 SMS/sec to Twilio
Level 4 - Global: max 10,000 requests/sec total
Request must pass ALL levels to proceed.
What is a token bucket and why per provider?
A token bucket refills at a fixed rate. Each request consumes a token. Empty = denied. Per-provider tracking enables informed failover.
package com.bank.ratelimit;
public class TokenBucketPerProvider {
public static void main(String[] args) {
System.out.println("Token bucket per provider:");
System.out.println(" Twilio bucket: capacity=200, refill=100/sec");
System.out.println(" MSG91 bucket: capacity=100, refill=50/sec");
System.out.println();
System.out.println(" Twilio bucket has 0 tokens -> EMPTY");
System.out.println(" MSG91 bucket has 45 tokens -> AVAILABLE");
System.out.println();
System.out.println(" Router asks each bucket: 'do you have a token?'");
System.out.println(" Twilio: NO -> skip");
System.out.println(" MSG91: YES -> route here");
System.out.println();
System.out.println(" Clean failover: informed, not blind");
}
}
Expected output:
Token bucket per provider:
Twilio bucket: 0 tokens -> EMPTY
MSG91 bucket: 45 tokens -> AVAILABLE
Router routes to MSG91 -> clean failover
What happens without per-provider tracking?
You hammer the exhausted provider, waste threads, and may get penalized.
package com.bank.ratelimit;
public class HammeringExhaustedProvider {
public static void main(String[] args) {
System.out.println("WITHOUT per-provider bucket:");
System.out.println(" Twilio rate limit exhausted");
System.out.println(" System keeps sending to Twilio");
System.out.println(" Every call rejected with 429");
System.out.println(" Threads wasted on rejected calls");
System.out.println(" Twilio may PENALIZE or BLOCK your account");
System.out.println(" Feedback loop: hammering worsens the situation");
System.out.println();
System.out.println("WITH per-provider bucket:");
System.out.println(" Twilio bucket empty -> do not even try");
System.out.println(" Route to MSG91 (still has tokens)");
System.out.println(" Zero wasted calls to Twilio");
}
}
Expected output:
WITHOUT per-provider bucket:
Twilio exhausted -> keep sending -> all rejected -> waste + penalty
WITH per-provider bucket:
Twilio empty -> skip -> route to MSG91 -> clean delivery
How do you implement multi-level in Spring?
Use Redis counters for per-user and per-IP, token buckets for per-provider, and a global counter at the gateway.
package com.bank.ratelimit;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.time.Duration;
@Service
public class MultiLevelLimiter {
private final StringRedisTemplate redis;
public MultiLevelLimiter(StringRedisTemplate redis) {
this.redis = redis;
}
public boolean isAllowed(String userId, String ip) {
// Level 1: Per-user
Long userCount = redis.opsForValue().increment("limit:user:" + userId);
if (userCount == 1) redis.expire("limit:user:" + userId, Duration.ofMinutes(1));
if (userCount > 5) return false;
// Level 2: Per-IP
Long ipCount = redis.opsForValue().increment("limit:ip:" + ip);
if (ipCount == 1) redis.expire("limit:ip:" + ip, Duration.ofMinutes(1));
if (ipCount > 20) return false;
// Level 3 & 4: Per-provider and global handled at gateway
return true;
}
}
Multi-level check:
Per-user: userId=rajesh -> count=3 -> PASS
Per-IP: ip=203.0.113.1 -> count=15 -> PASS
Per-provider: twilio bucket -> 50 tokens -> PASS
Global: total=8000/10000 -> PASS
All levels passed -> request proceeds
How do the levels compare?
| Level | Catches | Blind To |
|---|---|---|
| Per-user | One account monopolizing | Distributed attack across accounts |
| Per-IP | One source hitting many accounts | Distributed attack from many IPs |
| Per-provider | Exceeding downstream's rate | User-level abuse |
| Global | Any flood beyond total capacity | Which user or IP is causing it |
Composed together: each level's blind spot is another level's strength.
Removing any one opens a specific hole.
What is the relationship with circuit breakers?
Token bucket handles the normal rate limit. Circuit breaker handles a provider that is erroring or slow. They complement each other.
package com.bank.ratelimit;
public class BucketPlusBreaker {
public static void main(String[] args) {
System.out.println("Token bucket: tracks remaining allowance (rate limit)");
System.out.println("Circuit breaker: tracks error rate (health)");
System.out.println();
System.out.println("Scenario 1: Twilio is healthy but rate-limited");
System.out.println(" Bucket empty -> fail over to MSG91");
System.out.println();
System.out.println("Scenario 2: Twilio is returning 500 errors");
System.out.println(" Circuit breaker opens -> fail over to MSG91");
System.out.println();
System.out.println("Both together: skip provider if EMPTY or UNHEALTHY");
}
}
Expected output:
Token bucket: rate limit tracking
Circuit breaker: health tracking
Both together: skip if EMPTY or UNHEALTHY
What should a senior engineer know?
Size each bucket to real contracts. Monitor depletion as early warning. Implement in shared Redis. Prefer escalating responses.
Senior guidelines:
[x] Multi-level: per-user + per-IP + per-provider + global
[x] Token bucket per provider: tracks allowance independently
[x] Failover: route to provider with tokens when one empties
[x] Circuit breaker: complement bucket for health-based failover
[x] Redis: shared counters across all instances
[x] Monitor: bucket depletion as early-warning metric
[x] Escalate: CAPTCHA for per-IP, queue for global, 429 for per-user
Visual Overview
Multi-Level Rate Limiting:
Request
│
▼
┌──────────────┐ FAIL
│ Per-User │──────▶ 429 Too Many Requests
│ 5/min/acct │
└──────┬───────┘
PASS │
▼
┌──────────────┐ FAIL
│ Per-IP │──────▶ 429 + CAPTCHA
│ 20/min/IP │
└──────┬───────┘
PASS │
▼
┌──────────────┐ FAIL
│ Per-Provider │──────▶ Fail over to next provider
│ token bucket │
└──────┬───────┘
PASS │
▼
┌──────────────┐ FAIL
│ Global │──────▶ 503 Service Unavailable
│ 10000/sec │
└──────┬───────┘
PASS │
▼
Process Request
Per-Provider Failover:
┌────────────┐ empty ┌────────────┐ has tokens
│ Twilio │────────▶│ MSG91 │──────────▶ Send
│ bucket: 0 │ │ bucket: 45 │
└────────────┘ └────────────┘
Key Takeaway
Multi-level rate limiting beats a single limiter because each scope catches a pattern the others are blind to. Per-user stops one account monopolizing. Per-IP catches one source attacking many accounts. Per-provider keeps outbound calls within a downstream's rate. Global protects total capacity. Removing any one opens a specific hole, so you compose all levels.
The per-provider limit must be a token bucket per downstream because it tracks each provider's remaining allowance independently. When one provider's bucket empties, you immediately route to one that still has tokens instead of hammering the exhausted provider. Hammering wastes threads, triggers penalties, and can create a feedback loop. The bucket complements a circuit breaker: the bucket handles rate limits, the breaker handles errors.
At National Bank of India, a single global limiter on OTP sending failed both ways: an abusive script spread across many accounts slipped under, and lowering the threshold throttled real customers. Pankaj moved to layered limits with per-user, per-IP, global, and a token bucket per SMS provider in Redis. When the primary provider's bucket emptied, the router failed over to the secondary cleanly.
Remember one sentence: multi-level rate limiting composes per-user, per-IP, per-provider, and global scopes because each catches patterns the others miss, and a token bucket per provider enables clean failover to one with tokens instead of hammering an exhausted provider.
---