Definition
- Starvation problem – If OTP and marketing share one queue, millions of marketing messages sit ahead of the next OTP, ballooning its latency from seconds to minutes.
- Physical isolation – Separate Kafka topics for critical (OTP, security) and bulk (marketing), each with its own consumer group and worker fleet.
- Independent scaling – The critical group's progress is unaffected by the bulk group's lag; you can pause bulk without touching critical.
- Provider isolation – Reserve provider rate for the critical path with separate token buckets or credentials so starvation cannot occur at the last hop.
- Edge classification – Route each notification to the correct topic at enqueue time so marketing never enters the critical pipeline.
Imagine National Bank of India has two counters: a dedicated express counter for urgent OTP and security messages, and a separate bulk counter for marketing. The express counter has its own staff and its own phone line. Even when the bulk counter is overwhelmed or temporarily shut during an incident, the express counter keeps serving urgent messages instantly because it shares nothing with the bulk one.
Understanding the Concepts
Why does a shared queue cause starvation?
If OTP, security alerts, and marketing all flow through one topic and one consumer group, messages are processed in arrival order. When a campaign enqueues tens of millions of messages, those millions sit ahead of the next OTP. Even if consumers are fast, the sheer backlog means an OTP waits behind millions of messages, and a delayed OTP is a failed login.
package com.bank.notification.antipattern;
// WRONG: single shared topic for all priorities
public class SharedTopicAntipattern {
// Topic: "notifications"
// Consumer group: "notification-workers"
// Message order: [MARKETING, MARKETING, ...(10 million)..., OTP, MARKETING, ...]
// OTP at position 10,000,001 waits for 10 million marketing messages to drain.
// OTP delivery latency: seconds -> minutes. Login fails.
}
How does physical isolation fix this?
Create separate topics for critical (OTP, security) and bulk (marketing), each with its own consumer group.
package com.bank.notification.config;
public class TopicConfig {
public static final String CRITICAL_TOPIC = "notifications-critical";
public static final String BULK_TOPIC = "notifications-bulk";
// Critical consumer group: small, steady, always-on fleet
public static final String CRITICAL_GROUP = "critical-consumers";
// Bulk consumer group: large, elastic fleet
public static final String BULK_GROUP = "bulk-consumers";
// The two groups have independent offsets, independent lag,
// and independent scaling. Critical is never affected by bulk.
}
How do you classify and route at the edge?
Classify each notification by type and route to the correct topic at enqueue time.
package com.bank.notification.router;
public class NotificationRouter {
private final KafkaProducer producer;
public NotificationRouter(KafkaProducer producer) {
this.producer = producer;
}
public void route(Notification notification) {
String topic;
switch (notification.getType()) {
case OTP:
case SECURITY_ALERT:
case FRAUD_ALERT:
topic = TopicConfig.CRITICAL_TOPIC;
break;
case MARKETING:
case PROMOTIONAL:
topic = TopicConfig.BULK_TOPIC;
break;
default:
topic = TopicConfig.BULK_TOPIC; // safe default
}
producer.send(topic, notification.getUserId(), notification);
}
}
Why must provider capacity also be isolated?
If both pipelines share one provider rate limit, a marketing blast consumes the entire rate budget and the critical pipeline is starved at the final hop even with separate queues.
package com.bank.notification.provider;
public class IsolatedProviderRate {
// WRONG: shared rate limit
// TokenBucket shared = new TokenBucket(10_000); // both pipelines share
// CORRECT: reserved rate for critical
private final TokenBucket criticalBucket = new TokenBucket(2_000); // guaranteed
private final TokenBucket bulkBucket = new TokenBucket(8_000); // remainder
public boolean trySendCritical() {
return criticalBucket.tryConsume();
}
public boolean trySendBulk() {
return bulkBucket.tryConsume();
}
}
Why is a single priority queue weaker?
A priority queue helps but still shares consumers, provider rate, and failure modes. A poison message or provider throttle triggered by marketing volume can still affect OTP.
| Aspect | Single Priority Queue | Physical Isolation |
|---|---|---|
| Consumer threads | Shared | Independent per pipeline |
| Provider rate | Shared | Reserved per pipeline |
| Failure domain | Shared | Independent |
| Pause marketing | Cannot pause without pausing OTP | Pause bulk, keep critical |
| Independent scaling | No | Yes |
package com.bank.notification.antipattern;
// WRONG: single queue with priority flag
// A poison message in the shared consumer crashes it.
// Marketing volume triggers provider throttle -> OTP also throttled.
// Cannot pause marketing without pausing OTP.
// CORRECT: physical isolation (see above)
How do you pause bulk during incidents?
The bulk consumer group can be paused, throttled, or stopped while the critical group keeps running untouched.
package com.bank.notification.ops;
public class PipelineControl {
public void pauseMarketing() {
// Pause bulk consumer group: stop consuming from bulk topic
// Critical consumer group continues unaffected
System.out.println("Marketing pipeline PAUSED. OTP pipeline running.");
}
public void resumeMarketing() {
System.out.println("Marketing pipeline RESUMED.");
}
}
How do you monitor each pipeline?
Track critical-path latency and lag as a strict SLO with tight alerts. Bulk lag is expected to be large during campaigns.
package com.bank.notification.monitoring;
public class PipelineMonitor {
public void checkLatency(String pipeline, long lagMs) {
if ("critical".equals(pipeline) && lagMs > 5000) {
// ALERT: OTP latency above 5 seconds!
System.err.println("CRITICAL ALERT: OTP lag = " + lagMs + "ms");
}
if ("bulk".equals(pipeline) && lagMs > 3600_000) {
// Warning: marketing more than 1 hour behind
System.out.println("BULK WARNING: marketing lag = " + lagMs + "ms");
}
}
}
What are common interview follow-up questions?
How would you add a third priority tier for transactional alerts between critical and bulk? How do you handle a notification that was classified as marketing but is actually time-sensitive? What happens if the critical pipeline itself becomes overloaded? How do you share cost metrics between the two pipelines?
Visual Overview
┌──────────────────────────────────────────────────────────┐ │ PRIORITY ISOLATION FOR NOTIFICATIONS │ ├──────────────────────────────────────────────────────────┤ │ │ │ Incoming Notifications │ │ │ │ │ ┌──────▼───────────┐ │ │ │ Router/Classifier│ │ │ └────┬────────┬─────┘ │ │ │ │ │ │ OTP/Security Marketing │ │ │ │ │ │ ┌────▼─────┐ ┌────▼─────────────┐ │ │ │ CRITICAL │ │ BULK TOPIC │ │ │ │ TOPIC │ │ (millions of msgs)│ │ │ └────┬─────┘ └────┬─────────────┘ │ │ │ │ │ │ ┌────▼─────────┐ ┌────▼─────────────┐ │ │ │ Critical │ │ Bulk Consumers │ │ │ │ Consumers │ │ (elastic, can │ │ │ │ (small, │ │ be paused) │ │ │ │ steady) │ └────┬─────────────┘ │ │ └────┬─────────┘ │ │ │ │ │ │ │ ┌────▼──────┐ ┌─────▼──────┐ │ │ │ Reserved │ │ Remaining │ │ │ │ provider │ │ provider │ │ │ │ rate:2000 │ │ rate:8000 │ │ │ └───────────┘ └────────────┘ │ │ │ │ OTP latency: < 5 seconds (SLO) │ │ Marketing lag: hours during campaigns (expected) │ └──────────────────────────────────────────────────────────┘
Key Takeaway
At National Bank of India, OTP, security alerts, and marketing shared one Kafka topic and one consumer group. When a scheme campaign enqueued tens of millions of messages, Rajesh's OTP sat behind that backlog and its delivery latency went from seconds to minutes, meaning he could not log in. The root cause was that critical and bulk traffic shared the same queue, the same consumer threads, and the same provider rate budget.
Pankaj split the traffic into two physically isolated pipelines: a critical topic for OTP and security with a small, steady, always-on consumer group and reserved provider rate, and a bulk topic for marketing with a large, elastic consumer group and the remaining rate. Classification happened at the edge, so marketing could never enter the critical topic. After that, a million-message marketing blast had zero effect on OTP latency.
During one incident, the team paused the marketing pipeline entirely while OTP kept delivering in milliseconds, which would have been impossible with a shared queue. The provider rate was also isolated with separate token buckets so marketing volume could never consume the OTP rate budget at the last hop. The principle is that the only reliable way to guarantee the critical path under a bulk flood is to ensure it shares no resource with the flood.
Remember one sentence: Isolate priorities onto separate topics with dedicated consumer groups and reserved provider rate, so the critical OTP pipeline keeps low latency even when the marketing pipeline is saturated, paused, or experiencing failures.
---