Definition
- NotificationProvider interface – A single contract like
send(Notification)that hides the vendor (FCM, Twilio, SendGrid) from business logic. - Non-retryable error – A permanent failure (invalid token, hard bounce, HTTP 400) that will never succeed; mark failed immediately.
- Retryable error – A transient failure (timeout, HTTP 5xx, HTTP 429) that may succeed shortly; retry with backoff.
- Retry storm – Thousands of messages all retrying at the same instant, overwhelming a recovering provider.
- Exponential backoff with jitter – Each retry waits roughly double the previous delay plus a random spread to avoid synchronized waves.
Imagine National Bank of India sends alerts to Rajesh by SMS, email, and app push. Each courier company has a different procedure. Instead of the clerk learning three procedures, the bank defines one standard form. When a letter bounces, the clerk asks: did it bounce because the address does not exist (never retry) or because the courier's van broke down today (try again later)? Blindly resending to a courier whose vans are all broken only makes the traffic jam worse.
Understanding the Concepts
Why abstract providers behind an interface?
Vendor independence and testability. If your code imports com.twilio.rest.api.v2010.account.Message directly, every class is welded to Twilio.
package com.bank.notification;
public interface NotificationProvider {
DeliveryResult send(Notification notification);
NotificationChannel channel();
}
public enum NotificationChannel { SMS, EMAIL, PUSH }
public record Notification(String recipient, String message, NotificationChannel channel) {}
public record DeliveryResult(String messageId, DeliveryStatus status, String reason) {}
public enum DeliveryStatus { DELIVERED, FAILED_PERMANENT, FAILED_TRANSIENT }
package com.bank.notification;
import java.util.Map;
public class NotificationDispatcher {
private final Map<NotificationChannel, NotificationProvider> providers;
public NotificationDispatcher(Map<NotificationChannel, NotificationProvider> providers) {
this.providers = providers;
}
public DeliveryResult dispatch(Notification notification) {
NotificationProvider provider = providers.get(notification.channel());
if (provider == null) {
throw new IllegalArgumentException("No provider for " + notification.channel());
}
return provider.send(notification);
}
}
Benefits:
1. Vendor independence: swap Twilio for another SMS vendor -> change one class
2. Testability: inject FakeProvider in tests -> no real network calls
3. Failover: route to backup provider when primary is down
How do you classify errors?
Not all failures are equal. Non-retryable failures will never succeed. Retryable failures may succeed shortly.
package com.bank.notification;
public class ErrorClassification {
public static void main(String[] args) {
System.out.println("NON-RETRYABLE (permanent, never retry):");
System.out.println(" - Invalid FCM token (device unregistered)");
System.out.println(" - Hard email bounce (mailbox does not exist)");
System.out.println(" - Invalid phone number (not a valid MSISDN)");
System.out.println(" - HTTP 400 (malformed payload)");
System.out.println(" Action: mark FAILED, clean up bad token, do NOT retry");
System.out.println();
System.out.println("RETRYABLE (transient, retry with backoff):");
System.out.println(" - Connection timeout");
System.out.println(" - HTTP 500/502/503 (provider internal error)");
System.out.println(" - HTTP 429 (rate limited)");
System.out.println(" Action: retry with exponential backoff + jitter");
}
}
Expected output:
NON-RETRYABLE (permanent, never retry):
- Invalid FCM token (device unregistered)
- Hard email bounce (mailbox does not exist)
- Invalid phone number (not a valid MSISDN)
- HTTP 400 (malformed payload)
Action: mark FAILED, clean up bad token, do NOT retry
RETRYABLE (transient, retry with backoff):
- Connection timeout
- HTTP 500/502/503 (provider internal error)
- HTTP 429 (rate limited)
Action: retry with exponential backoff + jitter
What is a retry storm and how do you prevent it?
If 10,000 messages fail at the same instant and all retry 1 second later, the synchronized wave overwhelms the provider.
package com.bank.notification;
import java.util.concurrent.ThreadLocalRandom;
public class RetryStormPrevention {
public static long calculateDelay(int attempt, long baseMs, long capMs, long jitterMs) {
long exponential = Math.min(capMs, baseMs * (1L << attempt));
long jitter = ThreadLocalRandom.current().nextLong(0, jitterMs);
return exponential + jitter;
}
public static void main(String[] args) {
System.out.println("WRONG: retry all 10,000 after exactly 1 second");
System.out.println(" -> Synchronized wave hammers recovering provider");
System.out.println();
System.out.println("CORRECT: exponential backoff + jitter");
for (int attempt = 0; attempt < 4; attempt++) {
long delay = calculateDelay(attempt, 1000, 30000, 500);
System.out.printf(" Attempt %d: delay ~%dms%n", attempt + 1, delay);
}
System.out.println(" -> Retries spread across a window, not synchronized");
}
}
Expected output:
WRONG: retry all 10,000 after exactly 1 second
-> Synchronized wave hammers recovering provider
CORRECT: exponential backoff + jitter
Attempt 1: delay ~1234ms
Attempt 2: delay ~2456ms
Attempt 3: delay ~4123ms
Attempt 4: delay ~8345ms
-> Retries spread across a window, not synchronized
How do you handle HTTP 429 specifically?
Honour the Retry-After header. If throttling is sustained, open a per-provider circuit breaker.
package com.bank.notification;
public class Http429Handling {
public static void main(String[] args) {
System.out.println("HTTP 429 handling:");
System.out.println(" 1. Read Retry-After header (e.g. 30 seconds)");
System.out.println(" 2. Wait the specified duration before retrying");
System.out.println(" 3. If sustained 429s: open circuit breaker for this provider");
System.out.println(" -> Stop sending to exhausted provider");
System.out.println(" -> Route to backup provider if available");
System.out.println(" 4. Circuit half-opens after cooldown -> test recovery");
}
}
Expected output:
HTTP 429 handling:
1. Read Retry-After header (e.g. 30 seconds)
2. Wait the specified duration before retrying
3. If sustained 429s: open circuit breaker for this provider
4. Circuit half-opens after cooldown -> test recovery
What happens after max retries?
The message goes to a dead-letter queue (DLQ) for inspection. The DLQ is not a trash can.
package com.bank.notification;
public class DlqHandling {
public static void main(String[] args) {
System.out.println("After max retries (e.g. 5 attempts):");
System.out.println(" Message -> DLQ (dead-letter queue)");
System.out.println();
System.out.println("DLQ is NOT a trash can:");
System.out.println(" 1. Alert if DLQ volume is abnormal");
System.out.println(" 2. Inspect: why did this message fail?");
System.out.println(" 3. Fix the issue (provider config, data)");
System.out.println(" 4. Replay from DLQ after fix");
}
}
Expected output:
After max retries (e.g. 5 attempts):
Message -> DLQ (dead-letter queue)
DLQ is NOT a trash can:
1. Alert if DLQ volume is abnormal
2. Inspect: why did this message fail?
3. Fix the issue (provider config, data)
4. Replay from DLQ after fix
How does testability improve with the interface?
In tests, inject a FakeProvider that records what was sent. No real network calls.
package com.bank.notification;
import java.util.ArrayList;
import java.util.List;
public class FakeNotificationProvider implements NotificationProvider {
private final List<Notification> sentNotifications = new ArrayList<>();
@Override
public DeliveryResult send(Notification notification) {
sentNotifications.add(notification);
return new DeliveryResult("FAKE-001", DeliveryStatus.DELIVERED, "Fake delivery");
}
@Override
public NotificationChannel channel() {
return NotificationChannel.SMS;
}
public List<Notification> getSentNotifications() {
return sentNotifications;
}
}
Test assertion:
FakeProvider fake = new FakeProvider();
dispatcher.dispatch(new Notification("9876543210", "OTP: 123456", SMS));
assertEquals(1, fake.getSentNotifications().size());
assertEquals("OTP: 123456", fake.getSentNotifications().get(0).message());
// No real SMS sent. No Twilio bill.
What is the WRONG pattern vs CORRECT pattern?
package com.bank.notification;
public class WrongVsCorrect {
// WRONG: retry every failure the same way
// Priya's mistyped phone -> retried every minute FOREVER
// Wasting SMS quota on an invalid number
// CORRECT: classify first
// Invalid phone -> mark FAILED, clean up token, stop
// Timeout -> retry with backoff, max 5 attempts -> DLQ
public static void main(String[] args) {
System.out.println("WRONG: retry ALL errors equally");
System.out.println(" Invalid phone retried 100 times -> wasted quota");
System.out.println();
System.out.println("CORRECT: classify THEN decide");
System.out.println(" Invalid phone -> FAIL immediately, clean token");
System.out.println(" Timeout -> retry with backoff -> DLQ after max attempts");
}
}
Expected output:
WRONG: retry ALL errors equally
Invalid phone retried 100 times -> wasted quota
CORRECT: classify THEN decide
Invalid phone -> FAIL immediately, clean token
Timeout -> retry with backoff -> DLQ after max attempts
What should a senior engineer know?
Size provider buckets to real contracts. Monitor bucket depletion. Prefer escalating responses over hard rejects. Combine with per-provider circuit breaker.
Senior guidelines:
[x] Interface per provider (Dependency Inversion + Strategy pattern)
[x] Classify every error: permanent vs transient
[x] Permanent: fail fast, clean up bad tokens
[x] Transient: exponential backoff + jitter (never synchronized)
[x] HTTP 429: honour Retry-After + circuit breaker for sustained throttling
[x] DLQ after max retries: alert, inspect, replay
[x] FakeProvider for unit tests (no real network)
[x] Per-provider circuit breaker + token bucket for failover
Visual Overview
Notification Error Classification:
┌─────────────────┐
│ Send via │
│ Provider │
└────────┬────────┘
│
┌─────▼─────┐
│ Success? │
└──┬─────┬──┘
│YES │NO
▼ ▼
┌────────┐ ┌──────────────┐
│DELIVERED│ │ Classify │
└────────┘ │ error │
└──┬───────┬───┘
permanent│ │transient
▼ ▼
┌──────────────┐ ┌─────────────────────┐
│ FAIL │ │ Retry with │
│ immediately │ │ exponential backoff │
│ Clean token │ │ + jitter │
└──────────────┘ └─────────┬───────────┘
│ max retries
▼
┌──────────────────┐
│ DLQ (inspect, │
│ alert, replay) │
└──────────────────┘
Key Takeaway
Abstracting every notification vendor behind a NotificationProvider interface gives vendor independence, testability, and failover. The service depends on a stable contract like send(Notification) instead of a concrete SDK, so switching vendors means changing one implementation and one configuration line.
The harder and more important part is error classification. Not every failure should be retried. A non-retryable failure like an invalid token or a hard email bounce will never succeed no matter how many times you try; mark it failed immediately and clean up the bad token. A retryable failure like a timeout or HTTP 5xx may succeed shortly, but retrying naively is dangerous: if 10,000 messages all fail and retry at the same instant, the synchronized wave is a retry storm that keeps a recovering provider down.
At National Bank of India, Priya's mistyped phone number was retried every minute forever until Pankaj classified it as permanent and failed it on the first attempt. When Twilio had a 30-second blip, 5,000 alerts retried one second later and kept Twilio down. After introducing exponential backoff with jitter, the 5,000 retries spread smoothly and Twilio recovered. Every message that still fails after max attempts goes to a dead-letter queue for inspection and replay.
Remember one sentence: depend on a provider interface for vendor independence, always ask whether a failure is permanent or temporary before retrying, and use exponential backoff with jitter so retries spread across a window instead of becoming a synchronized storm.
---