Definition
- Decouple – A generator enqueues notification jobs; a separate worker fleet drains them at a sustainable rate. The queue is the shock absorber.
- Autoscale – Workers scale by queue depth: add when the backlog grows, remove when it drains.
- Rate limit – A token bucket on the outbound path ensures the fleet never exceeds the provider's allowed rate.
- Idempotency – Each notification has a unique ID; reprocessed jobs check and skip if already sent.
- Batch API – One request to Firebase carries hundreds of device tokens, reducing outbound calls dramatically.
- Progressive generation – For 100M recipients, paginate and enqueue in chunks rather than materializing all at once.
National Bank of India must text 100 million customers about a new scheme. If one clerk tried to call everyone at once, the branch collapses. Instead, one team writes names onto slips and drops them into a big in-tray (queue), and a separate team of senders takes slips and sends at a steady pace the phone network accepts. Each slip has a unique ticket number so nobody is messaged twice. The name-writers work through the list in chunks. Delivery confirmations arrive later and are ticked off.
Understanding the Concepts
What is the core problem?
Generating the recipient list (100 million rows) and delivering to the push provider are two different workloads at different rates. If you do them in one synchronous flow, generation overwhelms delivery. A durable queue between them converts the spiky process into a backpressured pipeline.
package com.bank.notification.model;
public class NotificationJob {
private String notificationId; // unique ID for idempotency
private String deviceToken;
private String title;
private String body;
private int priority; // 1=critical, 2=marketing
private long expiresAt;
public NotificationJob(String notificationId, String deviceToken,
String title, String body, int priority) {
this.notificationId = notificationId;
this.deviceToken = deviceToken;
this.title = title;
this.body = body;
this.priority = priority;
this.expiresAt = System.currentTimeMillis() + 3600_000; // 1 hour
}
public String getNotificationId() { return notificationId; }
public String getDeviceToken() { return deviceToken; }
public boolean isExpired() { return System.currentTimeMillis() > expiresAt; }
}
How does the generator produce jobs progressively?
Paginate the recipient set in chunks and enqueue a manageable number at a time. This keeps the queue bounded and lets you pause, resume, or cancel mid-flight.
package com.bank.notification.generator;
import java.util.List;
public class CampaignGenerator {
private final RecipientRepository recipientRepo;
private final NotificationQueue queue;
private static final int CHUNK_SIZE = 10_000;
public CampaignGenerator(RecipientRepository recipientRepo,
NotificationQueue queue) {
this.recipientRepo = recipientRepo;
this.queue = queue;
}
public void generateCampaign(String campaignId, String title, String body) {
long offset = 0;
List<Recipient> chunk;
do {
chunk = recipientRepo.findChunk(offset, CHUNK_SIZE);
for (Recipient r : chunk) {
String notifId = campaignId + "-" + r.getUserId();
queue.enqueue(new NotificationJob(
notifId, r.getDeviceToken(), title, body, 2));
}
offset += CHUNK_SIZE;
// Pace: wait if queue depth exceeds threshold
} while (chunk.size() == CHUNK_SIZE);
}
}
How do workers drain the queue with rate limiting?
Workers consume from the queue at whatever rate is sustainable. A token bucket on the outbound path ensures the fleet never exceeds the provider's limit.
package com.bank.notification.worker;
import com.bank.notification.model.NotificationJob;
import java.util.List;
import java.util.ArrayList;
public class DeliveryWorker implements Runnable {
private final NotificationQueue queue;
private final FirebaseClient firebaseClient;
private final IdempotencyStore idempotencyStore;
private final TokenBucket rateLimiter;
private static final int BATCH_SIZE = 500;
public DeliveryWorker(NotificationQueue queue, FirebaseClient firebaseClient,
IdempotencyStore idempotencyStore, TokenBucket rateLimiter) {
this.queue = queue;
this.firebaseClient = firebaseClient;
this.idempotencyStore = idempotencyStore;
this.rateLimiter = rateLimiter;
}
@Override
public void run() {
while (true) {
List<NotificationJob> batch = new ArrayList<>();
for (int i = 0; i < BATCH_SIZE; i++) {
NotificationJob job = queue.poll();
if (job == null) break;
if (job.isExpired()) continue; // discard stale
if (idempotencyStore.alreadySent(job.getNotificationId())) continue;
batch.add(job);
}
if (batch.isEmpty()) { sleep(100); continue; }
rateLimiter.acquire(); // wait for capacity
firebaseClient.sendBatch(batch); // 1 request -> 500 notifications
for (NotificationJob job : batch) {
idempotencyStore.markSent(job.getNotificationId());
}
}
}
private void sleep(long ms) {
try { Thread.sleep(ms); } catch (InterruptedException e) { }
}
}
How does the batch API reduce outbound calls?
Instead of one HTTP call per notification, one call carries 500 device tokens. This dramatically reduces request count and overhead.
package com.bank.notification.provider;
import com.bank.notification.model.NotificationJob;
import java.util.List;
import java.util.stream.Collectors;
public class FirebaseClient {
public void sendBatch(List<NotificationJob> batch) {
List<String> tokens = batch.stream()
.map(NotificationJob::getDeviceToken)
.collect(Collectors.toList());
// Firebase multicast: 1 HTTP request for up to 500 tokens
// Response includes per-token results: success, invalid, retry
System.out.printf("Sent batch of %d notifications in 1 request%n",
tokens.size());
}
}
Sent batch of 500 notifications in 1 request
# Without batching: 500 HTTP calls
# With batching: 1 HTTP call
# For 100M notifications at batch size 500: 200,000 requests instead of 100,000,000
How does idempotency prevent double-sending?
Each notification has a unique ID. Before sending, check if it was already delivered. After sending, record it. This turns at-least-once into effectively exactly-once.
package com.bank.notification.service;
import java.util.concurrent.ConcurrentHashMap;
public class IdempotencyStore {
private final ConcurrentHashMap<String, Boolean> sent = new ConcurrentHashMap<>();
public boolean alreadySent(String notificationId) {
return sent.containsKey(notificationId);
}
public void markSent(String notificationId) {
sent.put(notificationId, true);
}
}
How do retries and DLQ handle failures?
Transient failures retry with backoff. Permanently failed tokens (invalid, unregistered) go to a dead-letter queue for cleanup.
package com.bank.notification.worker;
public class RetryHandler {
private static final int MAX_RETRIES = 3;
public void handleFailure(NotificationJob job, String errorCode, int attempt) {
if ("INVALID_TOKEN".equals(errorCode) || "UNREGISTERED".equals(errorCode)) {
// Permanent failure: send to DLQ, clean up token store
deadLetterQueue.enqueue(job);
tokenCleanup.removeToken(job.getDeviceToken());
} else if (attempt < MAX_RETRIES) {
// Transient: re-enqueue with backoff
queue.enqueueWithDelay(job, attempt * 5000L);
} else {
deadLetterQueue.enqueue(job);
}
}
}
How do you track delivery?
The provider reports outcomes asynchronously through callbacks. Update each notification's state by unique ID.
package com.bank.notification.tracking;
public class DeliveryTracker {
public void onCallback(String notificationId, String status) {
// status: DELIVERED, OPENED, FAILED
System.out.printf("Notification %s: %s%n", notificationId, status);
// Update tracking store, aggregate campaign metrics
}
}
What are common interview follow-up questions?
How do you handle quiet hours and user notification preferences? How do you separate priority streams so OTPs are not stuck behind a marketing blast? How do you monitor queue depth and provider error rate? What happens if the provider rate-limits you despite your token bucket?
Visual Overview
┌────────────────────────────────────────────────────────┐ │ PUSH NOTIFICATION PIPELINE (100M) │ ├────────────────────────────────────────────────────────┤ │ │ │ Campaign Generator │ │ ┌────────────────────────┐ │ │ │ Paginate recipients │ │ │ │ in 10K chunks │ │ │ └──────────┬─────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────┐ │ │ │ Durable Queue (Kafka) │ ◀── shock absorber │ │ │ partitioned by user │ │ │ └──────────┬─────────────┘ │ │ │ │ │ ┌──────────▼─────────────┐ │ │ │ Worker Fleet │ ◀── autoscale by depth │ │ │ ┌───────────────────┐ │ │ │ │ │ Idempotency check │ │ │ │ │ │ Batch 500 tokens │ │ │ │ │ │ Rate limiter │ │ │ │ │ └───────┬───────────┘ │ │ │ └─────────┼──────────────┘ │ │ │ │ │ ┌─────────▼──────────────┐ │ │ │ Firebase / APNs │ │ │ │ (1 req → 500 tokens) │ │ │ └─────────┬──────────────┘ │ │ │ │ │ ┌─────────▼──────────────┐ ┌──────────────────┐ │ │ │ Async Callbacks │ │ DLQ: invalid │ │ │ │ (delivered/opened) │ │ tokens, cleanup │ │ │ └────────────────────────┘ └──────────────────┘ │ └────────────────────────────────────────────────────────┘
Key Takeaway
To send 100 million push notifications without overwhelming servers, the single most important decision is decoupling generation from delivery with a durable queue. At National Bank of India, an early attempt generated and sent synchronously, instantly overwhelming the senders and getting the system throttled by Firebase, with duplicate sends annoying customers after worker crashes.
Pankaj rebuilt it as a pipeline: a progressive generator feeding a partitioned Kafka queue in 10,000-recipient chunks, an autoscaled worker fleet that batched 500 device tokens per Firebase multicast request under a token-bucket rate limiter, a unique notification ID so reprocessed jobs skipped already-sent messages, and a dead-letter queue for invalid tokens that also fed token cleanup. After that, a 100-million campaign became a smooth background job that scaled the fleet up and back down.
Delivery tracking used Firebase's async callbacks, updating each notification's state (sent, delivered, opened, failed) by unique ID, giving precise campaign metrics without the send path ever blocking on confirmation. The nuance Pankaj adds is keeping a separate priority topic so time-critical OTPs are never stuck behind a marketing blast.
Remember one sentence: Decouple generation from delivery with a queue, autoscale workers by depth, rate-limit and batch to the provider, make each notification idempotent by unique ID, generate progressively, and track delivery via async callbacks.
---