Definition
- Message broker is a server that receives messages from producer applications and routes them to consumer applications using queues or topics.
- Queue is a channel where each message is delivered to exactly one consumer, following first-in-first-out (FIFO) order.
- Topic is a channel where each message is delivered to every active subscriber, enabling fan-out communication.
- Subscription is a consumer's registration on a topic, telling the broker to deliver a copy of every published message.
- Asynchronous communication means the producer sends a message and continues working without waiting for the consumer to process it.
Think of the National Bank of India branch office. When a customer submits a loan application at the counter, the clerk does not process it immediately. The clerk drops the application into a tray (the queue). A loan officer picks it up later and processes it. The customer does not wait at the counter. Now imagine the bank manager announces a new interest rate. That announcement goes on the notice board (the topic). Every department (home loans, car loans, personal loans) reads the same announcement independently. The tray is a queue: one person picks up each form. The notice board is a topic: everyone reads every notice.
Understanding the Concepts
What is a message broker?
A message broker is a dedicated server that sits between producers and consumers. The producer sends a message to the broker. The broker stores the message and delivers it to the right consumer when the consumer is ready.
Without a broker, Service A must call Service B directly over HTTP. If Service B is down, the call fails. With a broker in between, Service A drops the message into the broker and moves on. Service B picks it up when it comes back online.
The three main responsibilities of a message broker are:
- Decoupling. The producer does not need to know who the consumer is or where it runs.
- Buffering. Messages are stored until the consumer is ready, absorbing traffic spikes.
- Routing. The broker decides which consumer gets which message based on queues, topics, or routing rules.
package com.bank.messaging;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class SimpleBrokerDemo {
public static void main(String[] args) throws InterruptedException {
// A blocking queue simulates a simple message broker
BlockingQueue<String> loanQueue = new LinkedBlockingQueue<>();
// Producer: Rajesh submits a loan application
loanQueue.put("LOAN_APP:Rajesh:SAV2001:500000");
System.out.println("Producer: Submitted loan application for Rajesh");
// Consumer: Loan officer picks it up later
String message = loanQueue.take();
System.out.println("Consumer: Processing " + message);
}
}
Producer: Submitted loan application for Rajesh
Consumer: Processing LOAN_APP:Rajesh:SAV2001:500000
How does a queue work?
A queue follows point-to-point messaging. Each message goes to exactly one consumer. If three consumers listen on the same queue, the broker distributes messages round-robin so that each message is processed only once.
Queues are ideal when the message represents a task that should be done exactly once: process a loan, send an OTP, generate a statement.
- Messages are stored in FIFO order.
- Once a consumer acknowledges a message, it is removed from the queue.
- If the consumer crashes before acknowledging, the broker redelivers the message to another consumer.
package com.bank.messaging;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class QueueDemo {
public static void main(String[] args) throws InterruptedException {
BlockingQueue<String> transferQueue = new LinkedBlockingQueue<>();
// Three transfer requests from different customers
transferQueue.put("TXN:Rajesh:SAV2001->SAV3002:10000");
transferQueue.put("TXN:Priya:SAV3002->SAV4003:25000");
transferQueue.put("TXN:Amit:SAV5005->SAV2001:15000");
// Two consumers compete for messages
// Each message goes to exactly one consumer
System.out.println("Consumer-1 processes: " + transferQueue.take());
System.out.println("Consumer-2 processes: " + transferQueue.take());
System.out.println("Consumer-1 processes: " + transferQueue.take());
}
}
Consumer-1 processes: TXN:Rajesh:SAV2001->SAV3002:10000
Consumer-2 processes: TXN:Priya:SAV3002->SAV4003:25000
Consumer-1 processes: TXN:Amit:SAV5005->SAV2001:15000
How does a topic work?
A topic follows publish-subscribe messaging. Every subscriber receives a copy of every message. If three services subscribe to the "account-events" topic, all three get every account event.
Topics are ideal when the message represents an event that multiple services care about: account balance changed, loan approved, fraud detected.
- The producer publishes once.
- Each subscriber gets its own copy.
- Subscribers can join and leave independently without affecting each other.
package com.bank.messaging;
import java.util.List;
import java.util.ArrayList;
import java.util.function.Consumer;
public class TopicDemo {
private final List<Consumer<String>> subscribers = new ArrayList<>();
public void subscribe(String name, Consumer<String> handler) {
subscribers.add(msg -> {
System.out.println(name + " received: " + msg);
handler.accept(msg);
});
}
public void publish(String message) {
System.out.println("Published: " + message);
for (Consumer<String> sub : subscribers) {
sub.accept(message);
}
}
public static void main(String[] args) {
TopicDemo accountTopic = new TopicDemo();
// Three services subscribe to account events
accountTopic.subscribe("NotificationService", msg -> {});
accountTopic.subscribe("FraudDetectionService", msg -> {});
accountTopic.subscribe("StatementService", msg -> {});
// One event published, all three receive it
accountTopic.publish("BALANCE_CHANGED:SAV2001:Rajesh:75000");
}
}
Published: BALANCE_CHANGED:SAV2001:Rajesh:75000
NotificationService received: BALANCE_CHANGED:SAV2001:Rajesh:75000
FraudDetectionService received: BALANCE_CHANGED:SAV2001:Rajesh:75000
StatementService received: BALANCE_CHANGED:SAV2001:Rajesh:75000
How is a queue different from a topic?
| Feature | Queue | Topic |
|---|---|---|
| Delivery | Each message to one consumer | Each message to all subscribers |
| Use case | Task processing (process loan, send OTP) | Event broadcasting (balance changed, fraud alert) |
| Consumer count | Competing consumers share the load | Each subscriber gets every message |
| Message removal | Removed after acknowledgment | Retained for a configured retention period (Kafka) |
| Replay | Not possible after consumption | Possible by resetting offset (Kafka) |
| Scaling | Add consumers to process faster | Add subscribers for new features |
| Example | JMS Queue, SQS, RabbitMQ queue | Kafka topic, SNS, RabbitMQ exchange+fanout |
package com.bank.messaging;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.List;
import java.util.ArrayList;
import java.util.function.Consumer;
public class QueueVsTopicDemo {
public static void main(String[] args) throws InterruptedException {
System.out.println("=== Queue: Each message to one consumer ===");
BlockingQueue<String> loanQueue = new LinkedBlockingQueue<>();
loanQueue.put("LOAN:Rajesh:500000");
loanQueue.put("LOAN:Priya:300000");
// Only one consumer gets each message
System.out.println("Officer-A: " + loanQueue.take());
System.out.println("Officer-B: " + loanQueue.take());
System.out.println();
System.out.println("=== Topic: Each message to all subscribers ===");
List<Consumer<String>> subs = new ArrayList<>();
subs.add(msg -> System.out.println("SMS-Service: " + msg));
subs.add(msg -> System.out.println("Email-Service: " + msg));
subs.add(msg -> System.out.println("Audit-Service: " + msg));
String event = "FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer";
for (Consumer<String> sub : subs) {
sub.accept(event);
}
}
}
=== Queue: Each message to one consumer ===
Officer-A: LOAN:Rajesh:500000
Officer-B: LOAN:Priya:300000
=== Topic: Each message to all subscribers ===
SMS-Service: FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer
Email-Service: FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer
Audit-Service: FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer
How does Kafka combine queue and topic semantics?
Kafka uses a distributed commit log model that is different from traditional brokers. A Kafka topic is a log divided into partitions. Each partition is an ordered, append-only sequence of messages.
Kafka achieves queue behavior through consumer groups: all consumers in the same group share the partitions, so each message goes to one consumer in the group. Kafka achieves topic behavior by having multiple consumer groups: each group independently reads every message.
Key Kafka concepts:
- Topic. A named category of messages. Example:
account-transactions. - Partition. A segment of a topic. Partitions allow parallel processing.
- Offset. A sequential number assigned to each message in a partition.
- Consumer group. A set of consumers that cooperate to read from a topic, where each partition is assigned to exactly one consumer in the group.
- Broker. A Kafka server that stores partitions and serves read/write requests.
package com.bank.messaging;
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
public class KafkaTransactionProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// Key = account ID ensures all transactions for one account
// go to the same partition (ordering guarantee)
producer.send(new ProducerRecord<>(
"account-transactions", "SAV2001",
"{\"from\":\"SAV2001\",\"to\":\"SAV3002\",\"amount\":10000}"
));
System.out.println("Sent transaction event for SAV2001");
producer.send(new ProducerRecord<>(
"account-transactions", "SAV3002",
"{\"from\":\"SAV3002\",\"to\":\"SAV4003\",\"amount\":25000}"
));
System.out.println("Sent transaction event for SAV3002");
}
}
}
Sent transaction event for SAV2001
Sent transaction event for SAV3002
package com.bank.messaging;
import java.time.Duration;
import java.util.List;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
public class KafkaTransactionConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "transaction-processor");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("account-transactions"));
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
records.forEach(record ->
System.out.printf("Partition=%d Offset=%d Key=%s Value=%s%n",
record.partition(), record.offset(),
record.key(), record.value())
);
}
}
}
Partition=0 Offset=0 Key=SAV2001 Value={"from":"SAV2001","to":"SAV3002","amount":10000}
Partition=1 Offset=0 Key=SAV3002 Value={"from":"SAV3002","to":"SAV4003","amount":25000}
What are common mistakes with message brokers?
Mistake 1: Not handling duplicate messages
Wrong: assuming each message arrives exactly once.
package com.bank.messaging;
import java.math.BigDecimal;
public class NonIdempotentConsumer {
private BigDecimal balance = new BigDecimal("100000.00");
// WRONG: If the broker redelivers the message, the amount
// is credited twice. The customer gets double the money.
public void handleDeposit(String transactionId, BigDecimal amount) {
balance = balance.add(amount);
System.out.println("Balance after deposit: " + balance);
}
public static void main(String[] args) {
NonIdempotentConsumer consumer = new NonIdempotentConsumer();
// Same message delivered twice (broker retry after timeout)
consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
}
}
Balance after deposit: 150000.00
Balance after deposit: 200000.00
Correct: use an idempotency key to detect and skip duplicate messages.
package com.bank.messaging;
import java.math.BigDecimal;
import java.util.Set;
import java.util.HashSet;
public class IdempotentConsumer {
private BigDecimal balance = new BigDecimal("100000.00");
private final Set<String> processedIds = new HashSet<>();
// CORRECT: Check if we already processed this transaction.
// If yes, skip it. The balance stays correct.
public void handleDeposit(String transactionId, BigDecimal amount) {
if (processedIds.contains(transactionId)) {
System.out.println("Duplicate skipped: " + transactionId);
return;
}
processedIds.add(transactionId);
balance = balance.add(amount);
System.out.println("Balance after deposit: " + balance);
}
public static void main(String[] args) {
IdempotentConsumer consumer = new IdempotentConsumer();
consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
}
}
Balance after deposit: 150000.00
Duplicate skipped: TXN-1001
Mistake 2: Not setting a dead letter queue
Wrong: letting a poison message (one that always fails) block the entire queue forever.
Correct: after a configured number of retries, move the message to a dead letter queue for manual inspection.
package com.bank.messaging;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class DeadLetterQueueDemo {
static final int MAX_RETRIES = 3;
public static void main(String[] args) throws InterruptedException {
BlockingQueue<String> mainQueue = new LinkedBlockingQueue<>();
BlockingQueue<String> deadLetterQueue = new LinkedBlockingQueue<>();
mainQueue.put("TRANSFER:SAV2001:INVALID_ACCOUNT:-5000");
String message = mainQueue.take();
int retries = 0;
boolean processed = false;
while (retries < MAX_RETRIES && !processed) {
try {
processTransfer(message);
processed = true;
} catch (Exception e) {
retries++;
System.out.println("Retry " + retries + ": " + e.getMessage());
}
}
if (!processed) {
deadLetterQueue.put(message);
System.out.println("Moved to DLQ: " + message);
}
}
static void processTransfer(String msg) {
if (msg.contains("INVALID_ACCOUNT")) {
throw new IllegalArgumentException("Invalid target account");
}
}
}
Retry 1: Invalid target account
Retry 2: Invalid target account
Retry 3: Invalid target account
Moved to DLQ: TRANSFER:SAV2001:INVALID_ACCOUNT:-5000
How do you choose between synchronous and asynchronous communication?
| Criteria | Synchronous (HTTP/gRPC) | Asynchronous (Broker) |
|---|---|---|
| Response needed immediately | Yes: check balance, authenticate | No: send email, generate PDF |
| Coupling | Tight: caller waits for response | Loose: fire and forget |
| Failure handling | Caller retries or fails | Broker retries automatically |
| Throughput spikes | Caller overwhelms the callee | Broker buffers the spike |
| Ordering guarantee | Per-request ordering | Partition-level ordering (Kafka) |
| Debugging | Easier: single request/response | Harder: distributed trace needed |
In the National Bank of India system:
- Synchronous. Rajesh checks his account balance. The mobile app calls the Account Service over HTTP and waits for the response.
- Asynchronous. Priya transfers money. The Transfer Service publishes a
TransferInitiatedevent to Kafka. The Notification Service, Statement Service, and Fraud Detection Service each consume the event independently.
package com.bank.service;
import java.math.BigDecimal;
public class CommunicationChoiceDemo {
// Synchronous: caller waits for the result
static BigDecimal checkBalance(String accountId) {
System.out.println("[SYNC] Fetching balance for " + accountId);
return new BigDecimal("75000.00");
}
// Asynchronous: producer fires and forgets
static void initiateTransfer(String from, String to, BigDecimal amount) {
String event = String.format(
"TransferInitiated{from=%s, to=%s, amount=%s}", from, to, amount);
System.out.println("[ASYNC] Published: " + event);
// Producer does not wait for consumers to finish
}
public static void main(String[] args) {
// Rajesh checks balance: needs immediate answer
BigDecimal balance = checkBalance("SAV2001");
System.out.println("Rajesh balance: " + balance);
System.out.println();
// Priya transfers money: async is better
initiateTransfer("SAV3002", "SAV2001", new BigDecimal("25000.00"));
System.out.println("Priya continues without waiting");
}
}
[SYNC] Fetching balance for SAV2001
Rajesh balance: 75000.00
[ASYNC] Published: TransferInitiated{from=SAV3002, to=SAV2001, amount=25000.00}
Priya continues without waiting
What about message ordering and partitioning?
In Kafka, ordering is guaranteed only within a single partition. If you need all transactions for account SAV2001 to be processed in order, you use the account ID as the message key. Kafka hashes the key to determine the partition, so all messages with the same key go to the same partition.
If you do not set a key, Kafka distributes messages round-robin across partitions, and ordering is lost.
package com.bank.messaging;
import java.util.HashMap;
import java.util.Map;
import java.util.ArrayList;
import java.util.List;
public class PartitioningDemo {
// Simulate 3 partitions
private static final int PARTITION_COUNT = 3;
private static final Map<Integer, List<String>> partitions = new HashMap<>();
static {
for (int i = 0; i < PARTITION_COUNT; i++) {
partitions.put(i, new ArrayList<>());
}
}
static void send(String key, String value) {
int partition = Math.abs(key.hashCode()) % PARTITION_COUNT;
partitions.get(partition).add(key + ":" + value);
}
public static void main(String[] args) {
// All SAV2001 transactions go to the same partition
send("SAV2001", "deposit:10000");
send("SAV2001", "withdraw:5000");
send("SAV2001", "deposit:20000");
// SAV3002 transactions may go to a different partition
send("SAV3002", "deposit:30000");
send("SAV3002", "withdraw:10000");
for (Map.Entry<Integer, List<String>> entry : partitions.entrySet()) {
if (!entry.getValue().isEmpty()) {
System.out.println("Partition-" + entry.getKey()
+ ": " + entry.getValue());
}
}
}
}
Partition-0: [SAV3002:deposit:30000, SAV3002:withdraw:10000]
Partition-2: [SAV2001:deposit:10000, SAV2001:withdraw:5000, SAV2001:deposit:20000]
How do popular message brokers compare?
| Feature | Kafka | RabbitMQ | Amazon SQS | Apache Pulsar |
|---|---|---|---|---|
| Model | Distributed commit log | Traditional broker | Managed queue | Distributed log + queue |
| Ordering | Per-partition | Per-queue | Best-effort (FIFO available) | Per-partition |
| Retention | Time or size based | Until consumed | 4 days default (max 14) | Tiered storage |
| Replay | Yes (reset offset) | No | No | Yes |
| Throughput | Millions/sec | Thousands/sec | Thousands/sec | Millions/sec |
| Latency | Low (ms) | Very low (sub-ms) | Medium (10-50 ms) | Low (ms) |
| Delivery | At-least-once, exactly-once | At-least-once, at-most-once | At-least-once | At-least-once, exactly-once |
| Best for | Event streaming, audit logs | Task queues, RPC | Serverless, simple queues | Multi-tenant streaming |
package com.bank.messaging;
public class BrokerComparisonDemo {
// This code shows conceptual usage patterns, not real clients
public static void main(String[] args) {
// Kafka: event streaming for all account transactions
System.out.println("Kafka: publish to 'account-transactions' topic");
System.out.println(" - 3 partitions keyed by account ID");
System.out.println(" - Retained for 7 days for replay");
System.out.println(" - Consumer groups: processor, fraud, analytics");
System.out.println();
// RabbitMQ: task queue for statement generation
System.out.println("RabbitMQ: publish to 'statement-generation' queue");
System.out.println(" - 5 workers compete for messages");
System.out.println(" - Message removed after acknowledgment");
System.out.println(" - DLQ for failed generations");
System.out.println();
// SQS: serverless OTP delivery
System.out.println("SQS: publish to 'otp-delivery' queue");
System.out.println(" - Lambda triggers on new messages");
System.out.println(" - Visibility timeout: 30 seconds");
System.out.println(" - No infrastructure to manage");
}
}
Kafka: publish to 'account-transactions' topic
- 3 partitions keyed by account ID
- Retained for 7 days for replay
- Consumer groups: processor, fraud, analytics
RabbitMQ: publish to 'statement-generation' queue
- 5 workers compete for messages
- Message removed after acknowledgment
- DLQ for failed generations
SQS: publish to 'otp-delivery' queue
- Lambda triggers on new messages
- Visibility timeout: 30 seconds
- No infrastructure to manage
What about message serialization and schema evolution?
Messages need a format. Common choices are JSON, Avro, and Protobuf. In banking systems, schema evolution matters because producers and consumers are deployed independently.
Avro with a Schema Registry is the standard choice for Kafka in production. It validates schemas at write time and supports backward-compatible changes.
package com.bank.messaging;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;
import java.math.BigDecimal;
public class SerializationDemo {
record TransactionEvent(
String transactionId,
String fromAccount,
String toAccount,
BigDecimal amount,
String currency,
long timestamp
) {}
public static void main(String[] args) throws Exception {
ObjectMapper mapper = new ObjectMapper();
// Producer serializes
TransactionEvent event = new TransactionEvent(
"TXN-2001", "SAV2001", "SAV3002",
new BigDecimal("50000.00"), "INR",
System.currentTimeMillis()
);
String json = mapper.writeValueAsString(event);
System.out.println("Serialized: " + json);
// Consumer deserializes
JsonNode node = mapper.readTree(json);
System.out.println("Transaction ID: " + node.get("transactionId").asText());
System.out.println("Amount: " + node.get("amount").asText());
// Schema evolution: new field with default
// Old consumers ignore unknown fields
String newJson = json.replace("}",
",\"category\":\"NEFT\"}");
JsonNode newNode = mapper.readTree(newJson);
System.out.println("Category (new field): "
+ newNode.path("category").asText("UNKNOWN"));
}
}
Serialized: {"transactionId":"TXN-2001","fromAccount":"SAV2001","toAccount":"SAV3002","amount":50000.00,"currency":"INR","timestamp":1719500000000}
Transaction ID: TXN-2001
Amount: 50000.00
Category (new field): NEFT
What should a senior engineer know?
- Back-pressure. When consumers are slower than producers, messages pile up in the broker. Kafka handles this by letting consumers control their own polling rate. RabbitMQ uses prefetch limits to prevent overwhelming a consumer. In a banking system, back-pressure during month-end statement generation can cause millions of messages to queue up.
- Exactly-once semantics. Kafka supports exactly-once processing through idempotent producers (each message has a producer ID and sequence number) and transactional consumers (consume, process, produce, and commit as one atomic operation). For financial transactions, exactly-once is critical.
- Consumer lag monitoring. The difference between the latest offset in a partition and the consumer's committed offset is the consumer lag. If lag grows continuously, consumers cannot keep up. Monitor with Kafka's consumer-groups CLI or tools like Burrow.
- Message ordering across partitions. Kafka guarantees order only within a partition. If a banking workflow requires global ordering (rare), you must use a single partition, which limits throughput to one consumer.
- Compacted topics. Kafka supports log compaction where only the latest value for each key is retained. This is useful for maintaining a current snapshot of account balances without storing the full history.
- Idempotent consumers in production. Use a database table with a unique constraint on the message ID. Before processing, insert the ID. If the insert fails with a duplicate key, skip the message. This is more reliable than an in-memory Set.
What are common interview follow-up questions?
- What happens if a Kafka consumer crashes before committing its offset?
- How do you handle message ordering when a topic has multiple partitions?
- What is the difference between at-most-once, at-least-once, and exactly-once delivery?
- How do you implement a dead letter queue in Kafka?
- When would you choose RabbitMQ over Kafka?
- How do you monitor consumer lag in production?
- What is the Outbox pattern and why do you need it?
Visual Overview
Queue: Point-to-Point Messaging
================================
Producer Queue Consumers
──────── ────────── ────────────
Rajesh ──> │ MSG-1 │ ──> Consumer-A (processes MSG-1)
Priya ──> │ MSG-2 │ ──> Consumer-B (processes MSG-2)
Amit ──> │ MSG-3 │ ──> Consumer-A (processes MSG-3)
└────────┘
Each message goes to exactly ONE consumer.
Topic: Publish-Subscribe Messaging
===================================
Producer Topic Subscribers
──────── ────────── ──────────────────
┌─> NotificationService
Transfer ──> │ EVENT │ ──┼─> FraudDetectionService
Service └────────┘ └─> StatementService
Each subscriber gets a COPY of every message.
Kafka: Consumer Groups
========================
Topic: account-transactions (3 partitions)
┌──────────────────────────────────────────────┐
│ P0: [SAV2001 events] │
│ P1: [SAV3002 events] │
│ P2: [SAV4003 events] │
└──────────────────────────────────────────────┘
│ │
▼ ▼
Group: processor Group: fraud-detector
┌─────────────────┐ ┌──────────────────┐
│ C1 reads P0, P1 │ │ C3 reads P0 │
│ C2 reads P2 │ │ C4 reads P1, P2 │
└─────────────────┘ └──────────────────┘
Within a group: each partition to one consumer.
Across groups: every group reads all messages.
Key Takeaway
A message broker is the post office of a distributed system. When Rajesh submits a loan application at the National Bank of India, the clerk does not process it on the spot. The clerk drops it into a tray, and a loan officer picks it up when they are free. That tray is a queue. When the bank manager announces a new interest rate, every department reads the same notice. That notice board is a topic.
Queues and topics solve different problems. A queue ensures each message is processed exactly once by one consumer, which is perfect for tasks like generating monthly statements or sending OTP codes. A topic ensures every subscriber receives every message, which is perfect for events like account balance changes that the notification service, fraud detection, and statement service all need to see.
Kafka combines both patterns with consumer groups. Within a group, each partition goes to one consumer (queue behavior). Across groups, every group reads all messages (topic behavior). The partition key ensures ordering for related messages: all transactions for SAV2001 go to the same partition and are processed in order.
The most critical rule in financial messaging is idempotency. Brokers may redeliver messages after a timeout, a crash, or a network issue. If the consumer blindly processes every delivery, a deposit could be credited twice. Always track processed message IDs and skip duplicates.
One sentence to remember: a queue is one message to one consumer, a topic is one message to all subscribers, and Kafka's consumer groups give you both.
---