Definition
- Resend fix – The biggest single reduction: if a valid OTP already exists, re-send it instead of minting a new one, collapsing retry-storm amplification.
- Multi-level rate limiting – Per-user (3 OTPs in 5 minutes), per-IP (escalating to CAPTCHA), per-provider (token bucket at contractual rate), global cap.
- TTL queue – A priority queue with TTL absorbs the burst; expired send jobs are discarded rather than sending stale codes.
- Provider failover – Multiple SMS providers each with their own token bucket; fail over to the next provider that still has tokens.
- TOTP fallback – A time-based OTP generated on the user's device needs no network call and zero provider load.
National Bank of India's branch can only hand out ten thousand paper tokens per minute, but during a festival fifty thousand customers per second rush the counter. If every impatient customer who already has a valid token keeps re-asking for a new one, the counter is instantly overwhelmed. The single biggest fix is: if you already hold a valid token, the clerk re-hands you the same one, not a fresh one. Then add crowd control, a waiting line, a backup printer, and a mobile-app token that needs no printer at all.
Understanding the Concepts
What is the core problem?
The service receives 50,000 requests per second during a spike, but the SMS provider can only handle 10,000 messages per minute. You cannot forward every request; you must reduce the number of provider calls and pace the rest.
package com.bank.otp.model;
import java.time.Instant;
public class OtpRecord {
private String userId; // e.g. "CUST-RAJESH-001"
private String otpCode; // e.g. "482913"
private String phoneNumber;
private Instant createdAt;
private Instant expiresAt;
private int sendCount; // how many times this OTP was sent
public OtpRecord(String userId, String otpCode, String phoneNumber,
int ttlSeconds) {
this.userId = userId;
this.otpCode = otpCode;
this.phoneNumber = phoneNumber;
this.createdAt = Instant.now();
this.expiresAt = createdAt.plusSeconds(ttlSeconds);
this.sendCount = 1;
}
public boolean isValid() {
return Instant.now().isBefore(expiresAt);
}
public String getOtpCode() { return otpCode; }
public void incrementSendCount() { sendCount++; }
}
How does the Resend fix work?
When a user taps Resend, check if a valid OTP already exists. If yes, re-send it. If no, generate a new one. This collapses 5 taps into 1 generation plus 4 cheap re-sends.
package com.bank.otp.service;
import com.bank.otp.model.OtpRecord;
import java.security.SecureRandom;
public class OtpService {
private final OtpStore otpStore; // Redis-backed
private final SmsGateway smsGateway;
private final SecureRandom random = new SecureRandom();
public OtpService(OtpStore otpStore, SmsGateway smsGateway) {
this.otpStore = otpStore;
this.smsGateway = smsGateway;
}
public String requestOtp(String userId, String phoneNumber) {
// Check for existing valid OTP
OtpRecord existing = otpStore.getActiveOtp(userId);
if (existing != null && existing.isValid()) {
// Resend the SAME OTP, do NOT generate a new one
existing.incrementSendCount();
smsGateway.send(phoneNumber, "Your OTP is: " + existing.getOtpCode());
return existing.getOtpCode();
}
// Generate new OTP
String code = String.format("%06d", random.nextInt(1_000_000));
OtpRecord record = new OtpRecord(userId, code, phoneNumber, 300);
otpStore.save(userId, record); // Redis SET with 300s TTL
smsGateway.send(phoneNumber, "Your OTP is: " + code);
return code;
}
}
# Rajesh taps "Send OTP" -> generates 482913, sends SMS (1 provider call)
# Rajesh taps "Resend" -> existing 482913 is valid, re-sends (1 cheap call)
# Rajesh taps "Resend" again -> same 482913 (1 cheap call)
# Without the fix: 3 taps = 3 new OTPs = 3 full provider calls
# With the fix: 3 taps = 1 new OTP + 2 re-sends (rate-limited)
How does multi-level rate limiting work?
Each level catches a different abuse pattern. The per-provider token bucket is what makes failover work.
package com.bank.otp.ratelimit;
import java.util.concurrent.atomic.AtomicInteger;
public class MultiLevelRateLimiter {
// Per-user: max 3 OTPs in 5 minutes
public boolean checkUserLimit(String userId) {
int count = getUserOtpCount(userId);
return count < 3;
}
// Per-IP: max 10 OTPs in 1 minute (escalate to CAPTCHA)
public boolean checkIpLimit(String ipAddress) {
int count = getIpOtpCount(ipAddress);
return count < 10;
}
// Per-provider: token bucket at contractual rate
public boolean checkProviderLimit(String providerId) {
return getProviderBucket(providerId).tryConsume();
}
// Global: max 100,000 OTPs system-wide per minute
public boolean checkGlobalLimit() {
return getGlobalBucket().tryConsume();
}
// Stubs for brevity
private int getUserOtpCount(String u) { return 0; }
private int getIpOtpCount(String i) { return 0; }
private TokenBucket getProviderBucket(String p) { return new TokenBucket(); }
private TokenBucket getGlobalBucket() { return new TokenBucket(); }
}
How does the TTL-and-priority queue absorb the burst?
Instead of calling the provider synchronously, enqueue send jobs. Workers drain at the provider's rate. Expired jobs are discarded.
package com.bank.otp.queue;
import java.time.Instant;
public class OtpSendJob {
private String phoneNumber;
private String message;
private int priority; // 1=high-value transfer, 2=login
private Instant expiresAt; // discard if past this time
public boolean isExpired() {
return Instant.now().isAfter(expiresAt);
}
public int getPriority() { return priority; }
}
// Worker:
// while (true) {
// OtpSendJob job = queue.poll(); // priority queue
// if (job.isExpired()) { discard; continue; }
// providerClient.send(job); // paced at provider's rate
// }
How does provider failover work?
Multiple providers, each with its own token bucket. When one is exhausted or unhealthy, route to the next.
package com.bank.otp.provider;
import java.util.List;
public class FailoverSmsGateway {
private final List<SmsProvider> providers; // ordered: primary, secondary
public FailoverSmsGateway(List<SmsProvider> providers) {
this.providers = providers;
}
public boolean send(String phoneNumber, String message) {
for (SmsProvider provider : providers) {
if (provider.isHealthy() && provider.hasCapacity()) {
boolean sent = provider.send(phoneNumber, message);
if (sent) return true;
}
}
return false; // all providers exhausted
}
}
What about TOTP as a provider-independent fallback?
A time-based OTP is generated on the user's device from a shared secret and a clock. It needs no SMS, no network call, and zero provider load.
package com.bank.otp.totp;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.ByteBuffer;
public class TotpGenerator {
private static final int PERIOD = 30; // seconds
private static final int DIGITS = 6;
public static String generateTotp(byte[] secret) throws Exception {
long timeStep = System.currentTimeMillis() / 1000 / PERIOD;
byte[] timeBytes = ByteBuffer.allocate(8).putLong(timeStep).array();
Mac mac = Mac.getInstance("HmacSHA1");
mac.init(new SecretKeySpec(secret, "HmacSHA1"));
byte[] hash = mac.doFinal(timeBytes);
int offset = hash[hash.length - 1] & 0x0F;
int code = ((hash[offset] & 0x7F) << 24
| (hash[offset + 1] & 0xFF) << 16
| (hash[offset + 2] & 0xFF) << 8
| (hash[offset + 3] & 0xFF)) % (int) Math.pow(10, DIGITS);
return String.format("%06d", code);
}
}
# TOTP for Rajesh: 738291 (valid for 30 seconds)
# No SMS needed. No provider call. Zero load on the spike.
What are common interview follow-up questions?
How would you handle the case where the user's TOTP secret is compromised? How do you prevent OTP brute-force attacks? What is the maximum number of verification attempts you allow? How do you handle users who cannot receive SMS in certain countries? How does the Resend cooldown interact with the per-user rate limit?
What should a senior engineer know?
A senior engineer measures the actual retry-storm multiplier before and after the Resend fix, monitors per-provider token-bucket fill rates, and sets up alerts for when the primary provider nears capacity so failover activates before total exhaustion. They also audit OTP brute-force attempts with a maximum verification-attempt limit per OTP.
package com.bank.otp;
public class SeniorOtpGuidelines {
public static void main(String[] args) {
System.out.println("Senior OTP guidelines:");
System.out.println(" 1. Measure retry-storm multiplier: N taps / N provider calls");
System.out.println(" 2. Max verification attempts per OTP: 3 (block after 3 wrong)");
System.out.println(" 3. Alert when primary provider bucket < 20% capacity");
System.out.println(" 4. Audit log every OTP request: userId, IP, provider, timestamp");
System.out.println(" 5. TOTP enrollment reduces provider dependency permanently");
System.out.println(" 6. Never log or cache the OTP code in plain text");
System.out.println(" 7. Use SecureRandom, not Random, for OTP generation");
}
}
Expected output:
Senior OTP guidelines:
1. Measure retry-storm multiplier: N taps / N provider calls
2. Max verification attempts per OTP: 3 (block after 3 wrong)
3. Alert when primary provider bucket < 20% capacity
4. Audit log every OTP request: userId, IP, provider, timestamp
5. TOTP enrollment reduces provider dependency permanently
6. Never log or cache the OTP code in plain text
7. Use SecureRandom, not Random, for OTP generation
Visual Overview
┌─────────────────────────────────────────────────────────┐ │ OTP SERVICE UNDER SPIKE │ ├─────────────────────────────────────────────────────────┤ │ │ │ User taps "Send OTP" or "Resend" │ │ │ │ │ ▼ │ │ ┌──────────────────────┐ │ │ │ Rate Limiter │ │ │ │ (user, IP, global) │──REJECT──▶ 429 Too Many │ │ └──────────┬───────────┘ │ │ │ PASS │ │ ▼ │ │ ┌──────────────────────┐ │ │ │ Existing valid OTP? │ │ │ └────┬─────────┬───────┘ │ │ YES │ │ NO │ │ │ ┌────▼─────────┐ │ │ Re-send │ Generate new │ │ │ same OTP │ OTP + Redis │ │ │ │ └────┬─────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌──────────────────────┐ │ │ │ TTL + Priority Queue │ │ │ │ (expired = discard) │ │ │ └──────────┬───────────┘ │ │ │ │ │ ┌──────────▼───────────────────────────┐ │ │ │ Provider Failover │ │ │ │ ┌──────────┐ ┌──────────┐ ┌────────┐ │ │ │ │ │Provider A│ │Provider B│ │ TOTP │ │ │ │ │ │(token │ │(token │ │(no SMS)│ │ │ │ │ │ bucket) │ │ bucket) │ │ │ │ │ │ │ └──────────┘ └──────────┘ └────────┘ │ │ │ └──────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘
Key Takeaway
An OTP service that survives a login spike is fundamentally a demand-supply mismatch problem. At National Bank of India, a festival promotion produced 50,000 OTP requests per second onto an SMS provider that could handle 10,000 per minute. The single most effective fix was making the Resend button re-send the existing valid OTP instead of generating a new one. Impatient users like Rajesh tapping Resend five times had been generating five full provider calls; after the fix, one OTP was generated and the re-sends were cheap, paced operations that collapsed the retry storm.
On top of that, Pankaj layered multi-level rate limiting: per-user (3 in 5 minutes), per-IP (escalating to CAPTCHA), per-provider (token bucket at the contractual rate), and a global cap. A TTL-and-priority queue absorbed the burst and drained at the provider's sustainable rate, discarding expired jobs that would have sent stale codes. High-value actions like Priya's large transfer jumped ahead of routine logins in the priority queue.
Provider failover routed to the next provider that still had capacity when the primary's token bucket was empty, and TOTP for enrolled customers like Pankaj let them log in with zero provider involvement. The through-line is: reduce provider calls first (Resend fix), then pace and route what remains (queue, failover), then fall back to channels that need no provider (TOTP, email).
Remember one sentence: Survive an OTP spike by making Resend re-send the existing valid OTP to kill retry-storm amplification, then layer multi-level rate limiting, a TTL-and-priority queue, multi-provider failover, and TOTP as the provider-independent fallback.
---