Definition
- Kafka – a distributed, append-only, durable event log built for high-throughput streaming where many independent consumer groups read the same events and can replay history.
- AWS SQS – a fully managed cloud queue built for simple point-to-point job distribution with no brokers to operate, offering standard (at-least-once, best-effort ordering) and FIFO (exactly-once, strict ordering) variants.
- RabbitMQ – a traditional AMQP message broker with exchanges, routing keys, and bindings that give fine-grained control over which queue receives which message.
- Consumer group (Kafka) – a set of consumers sharing a group id that cooperate to consume a topic, enabling parallel processing where each record is handled once per group.
- Exchange (RabbitMQ) – a routing component that receives messages from producers and distributes them to bound queues based on routing keys and exchange type (direct, topic, fanout).
- Visibility timeout (SQS) – the period during which a message being processed is hidden from other consumers, after which it reappears if not deleted.
Imagine National Bank of India needs to move information around. Kafka is the bank's permanent transaction ledger: every event like "Rajesh transferred 50000 rupees" is written down forever, and the fraud team, the audit team, and the notification team each read the same ledger at their own pace, even re-reading last week's entries. SQS is a basket of work slips on a clerk's desk: a background clerk picks one slip, does the job, and throws the slip away. RabbitMQ is the bank's internal mail room with labeled pigeonholes: a letter marked "loan.approval.mumbai" is routed only to the Mumbai loan desk based on its label.
Understanding the Concepts
What is Kafka and when do you choose it?
Kafka is a distributed commit log where producers append events to partitioned topics. The defining property is that Kafka retains messages based on a time or size policy, not based on whether they were consumed. Multiple independent consumer groups can read the same events, and a consumer can rewind its offset to re-read old events. This makes Kafka ideal for event streaming, event sourcing, and any scenario where multiple systems need the same data.
At National Bank of India, Rajesh's transfer event is published once to the transfer-events topic, and the fraud service, audit service, and notification service each consume it independently. If the fraud team needs to replay last week's events to retrain a model, they simply seek back to an earlier offset.
package com.bank.producer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class TransferEventProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
public TransferEventProducer(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishTransfer(String accountId, BigDecimal amount) {
String event = "{\"accountId\":\"" + accountId
+ "\",\"amount\":" + amount + "}";
kafkaTemplate.send("transfer-events", accountId, event);
System.out.println("Published transfer event for: " + accountId);
}
}
Expected output:
Published transfer event for: SAV2001
-> Fraud service reads it (fraud-group)
-> Audit service reads it (audit-group)
-> Notification service reads it (notification-group)
What is AWS SQS and when do you choose it?
SQS is a pull-based, fully managed queue. A message lives in the queue until a consumer deletes it or the retention window (up to 14 days) expires. When a consumer reads a message, it becomes invisible for a visibility timeout; if the consumer crashes before deleting it, the message reappears for another worker. SQS is ideal when exactly one worker should perform a background job, and you want zero infrastructure to manage.
At National Bank of India, when a monthly statement needs to be generated as a PDF, the system puts a job message into an SQS queue. One worker picks it up, generates the PDF, and deletes the message. No Kafka infrastructure needed for this simple background task.
package com.bank.sqs;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.SendMessageRequest;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest;
import software.amazon.awssdk.services.sqs.model.DeleteMessageRequest;
import software.amazon.awssdk.services.sqs.model.Message;
import java.util.List;
public class StatementJobQueue {
private final SqsClient sqsClient;
private final String queueUrl;
public StatementJobQueue(SqsClient sqsClient, String queueUrl) {
this.sqsClient = sqsClient;
this.queueUrl = queueUrl;
}
public void enqueueStatementJob(String accountId, String month) {
String body = "{\"accountId\":\"" + accountId
+ "\",\"month\":\"" + month + "\"}";
sqsClient.sendMessage(SendMessageRequest.builder()
.queueUrl(queueUrl).messageBody(body).build());
System.out.println("Enqueued statement job: " + accountId);
}
public void processJobs() {
List<Message> messages = sqsClient.receiveMessage(
ReceiveMessageRequest.builder()
.queueUrl(queueUrl).maxNumberOfMessages(5).build()
).messages();
for (Message msg : messages) {
System.out.println("Generating statement: " + msg.body());
// Generate PDF...
sqsClient.deleteMessage(DeleteMessageRequest.builder()
.queueUrl(queueUrl)
.receiptHandle(msg.receiptHandle()).build());
System.out.println("Job completed and deleted");
}
}
}
Expected output:
Enqueued statement job: SAV2001
Generating statement: {"accountId":"SAV2001","month":"2025-01"}
Job completed and deleted
What is RabbitMQ and when do you choose it?
RabbitMQ is a smart broker implementing the AMQP protocol. Producers publish to an exchange, not directly to a queue. The exchange uses a routing key and bindings to decide which queue receives the message. A direct exchange matches the routing key exactly, a topic exchange matches wildcard patterns like loan.*.mumbai, and a fanout exchange broadcasts to all bound queues. RabbitMQ is excellent for complex routing logic that changes frequently.
At National Bank of India, loan applications need routing based on region and loan type. A topic exchange with routing keys like loan.home.mumbai routes the application to the Mumbai home loan desk, while loan.personal.delhi goes to the Delhi personal loan desk.
package com.bank.rabbitmq;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.stereotype.Service;
@Service
public class LoanApplicationRouter {
private final RabbitTemplate rabbitTemplate;
public LoanApplicationRouter(RabbitTemplate rabbitTemplate) {
this.rabbitTemplate = rabbitTemplate;
}
public void routeLoanApplication(String loanType, String region,
String applicationJson) {
String routingKey = "loan." + loanType + "." + region;
rabbitTemplate.convertAndSend("loan-exchange", routingKey,
applicationJson);
System.out.println("Routed loan application: " + routingKey);
}
}
Expected output:
Routed loan application: loan.home.mumbai
-> Delivered to mumbai-home-loan-queue (bound to loan.home.mumbai)
Routed loan application: loan.personal.delhi
-> Delivered to delhi-personal-loan-queue (bound to loan.personal.delhi)
How are Kafka, SQS, and RabbitMQ different? (comparison table)
The differences are rooted in the core design philosophy. Kafka is a log, SQS is a queue, and RabbitMQ is a router.
| Feature | Kafka | AWS SQS | RabbitMQ |
|---|---|---|---|
| Core model | Append-only event log | Managed point-to-point queue | Smart broker with exchanges |
| Message retention | Time/size based, independent of consumption | Until consumed or 14 days max | Until consumed and acknowledged |
| Multiple consumers | Yes, via consumer groups (each gets all) | No, one consumer per message | Yes, via fanout exchange |
| Replay | Yes, seek to any offset | No, message deleted after ack | No, message removed after ack |
| Ordering | Per partition (by key) | Best-effort (Standard), strict (FIFO) | Per queue |
| Routing | Topic + key-based partitioning | None (simple queue) | Exchanges, routing keys, bindings |
| Throughput | Millions of events/second | Thousands/second (scales with SQS) | Tens of thousands/second |
| Operations | Self-managed cluster | Zero (fully managed by AWS) | Self-managed broker |
| Delivery | At-least-once, exactly-once with transactions | At-least-once (Standard), exactly-once (FIFO) | At-least-once, at-most-once |
| Best for | Event streaming, event sourcing, CDC | Background jobs, spike buffering | Complex routing, request-reply |
package com.bank.comparison;
public class MessagingDecisionMatrix {
public static String chooseSystem(boolean needsReplay,
boolean needsRouting,
boolean zeroOps) {
if (needsReplay) {
return "Kafka: durable log with replay capability";
}
if (needsRouting) {
return "RabbitMQ: exchange-based routing";
}
if (zeroOps) {
return "SQS: fully managed, zero infrastructure";
}
return "Kafka: default for event-driven architecture";
}
public static void main(String[] args) {
System.out.println(chooseSystem(true, false, false));
System.out.println(chooseSystem(false, true, false));
System.out.println(chooseSystem(false, false, true));
}
}
Expected output:
Kafka: durable log with replay capability
RabbitMQ: exchange-based routing
SQS: fully managed, zero infrastructure
What are the common mistakes when choosing a messaging system?
The most common mistake is using one system for everything when a different system would be a better fit. Another mistake is choosing Kafka for simple background jobs where SQS would be simpler and cheaper.
Wrong approach: using Kafka for a simple email notification job.
package com.bank.messaging;
// WRONG: Using Kafka for a simple one-off background job.
// Kafka is overkill here. You maintain a Kafka cluster, configure
// topics, partitions, consumer groups, and retention for a task
// that just needs one worker to send one email.
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class WrongEmailNotifier {
@KafkaListener(topics = "email-jobs", groupId = "email-group")
public void sendEmail(String payload) {
// Simple email sending - does not need replay, multiple consumers,
// or durable event history
System.out.println("Sending email: " + payload);
}
}
Correct approach: using SQS for the simple job, Kafka for the event stream.
package com.bank.messaging;
// CORRECT: SQS for simple background jobs, Kafka for event streams.
// SQS is fully managed, no cluster to maintain, perfect for
// one-worker-per-job patterns.
public class CorrectSystemChoice {
// Use Kafka for the event that triggered the notification
// kafkaTemplate.send("transfer-completed", accountId, event);
// Use SQS for the actual email sending job
// sqsClient.sendMessage(queueUrl, emailPayload);
public static void main(String[] args) {
System.out.println("Kafka: publish transfer-completed event");
System.out.println(" -> Fraud service reads event (Kafka consumer)");
System.out.println(" -> Audit service reads event (Kafka consumer)");
System.out.println("SQS: enqueue email job");
System.out.println(" -> One worker sends email and deletes job");
}
}
Expected output:
Kafka: publish transfer-completed event
-> Fraud service reads event (Kafka consumer)
-> Audit service reads event (Kafka consumer)
SQS: enqueue email job
-> One worker sends email and deletes job
What about using all three together in a banking system?
A mature banking platform at National Bank of India uses all three systems, each for the workload it handles best. Kafka is the event backbone for all financial events. SQS handles background jobs like report generation and PDF creation. RabbitMQ handles legacy routing-heavy integrations where messages need to reach specific desks based on complex rules.
package com.bank.architecture;
public class MessagingArchitectureOverview {
public static void main(String[] args) {
System.out.println("National Bank of India Messaging Architecture");
System.out.println("============================================");
System.out.println();
System.out.println("Kafka (Event Backbone):");
System.out.println(" - transfer-events: all money movements");
System.out.println(" - trade-events: stock trading activity");
System.out.println(" - audit-events: compliance logging");
System.out.println(" Consumers: fraud, audit, notification, analytics");
System.out.println();
System.out.println("SQS (Background Jobs):");
System.out.println(" - statement-generation-queue: monthly PDFs");
System.out.println(" - report-queue: regulatory reports");
System.out.println(" - email-queue: customer notifications");
System.out.println();
System.out.println("RabbitMQ (Routing):");
System.out.println(" - loan-exchange: route by type and region");
System.out.println(" - support-exchange: route by priority and dept");
}
}
Expected output:
National Bank of India Messaging Architecture
============================================
Kafka (Event Backbone):
- transfer-events: all money movements
- trade-events: stock trading activity
- audit-events: compliance logging
Consumers: fraud, audit, notification, analytics
SQS (Background Jobs):
- statement-generation-queue: monthly PDFs
- report-queue: regulatory reports
- email-queue: customer notifications
RabbitMQ (Routing):
- loan-exchange: route by type and region
- support-exchange: route by priority and dept
What about throughput and scalability differences?
Kafka handles millions of events per second because it appends to a sequential log on disk and batches writes. SQS scales automatically but typically handles thousands of messages per second per queue. RabbitMQ handles tens of thousands per second but is limited by single-node memory and the complexity of exchange routing.
package com.bank.benchmark;
public class ThroughputComparison {
public static void main(String[] args) {
System.out.println("Throughput Comparison (approximate):");
System.out.println("Kafka: 1,000,000+ events/sec (batched, sequential I/O)");
System.out.println("SQS: 3,000 msg/sec (standard), 300 msg/sec (FIFO)");
System.out.println("RabbitMQ: 20,000-50,000 msg/sec (depends on routing)");
System.out.println();
System.out.println("Banking recommendation:");
System.out.println(" High-volume trade feed -> Kafka");
System.out.println(" Statement generation jobs -> SQS");
System.out.println(" Loan routing to branches -> RabbitMQ");
}
}
Expected output:
Throughput Comparison (approximate):
Kafka: 1,000,000+ events/sec (batched, sequential I/O)
SQS: 3,000 msg/sec (standard), 300 msg/sec (FIFO)
RabbitMQ: 20,000-50,000 msg/sec (depends on routing)
Banking recommendation:
High-volume trade feed -> Kafka
Statement generation jobs -> SQS
Loan routing to branches -> RabbitMQ
What are common interview follow-up questions?
Interviewers often ask why you would not just use Kafka for everything. The answer is that Kafka adds operational complexity: you manage a broker cluster, configure partitions, manage consumer groups, and handle retention. For a simple background job, SQS gives you the same reliability with zero ops. They also ask when RabbitMQ beats Kafka. The answer is complex routing: RabbitMQ's exchange-binding model handles patterns like loan.*.mumbai natively, while Kafka would require multiple topics or application-level filtering.
package com.bank.interview;
public class InterviewFollowUp {
public static void main(String[] args) {
System.out.println("Q: Why not use Kafka for everything?");
System.out.println("A: Kafka needs a cluster, partitions, consumer groups.");
System.out.println(" SQS is zero-ops for simple jobs.");
System.out.println();
System.out.println("Q: When does RabbitMQ beat Kafka?");
System.out.println("A: Complex routing with exchanges and wildcard keys.");
System.out.println(" Kafka requires app-level filtering for the same.");
System.out.println();
System.out.println("Q: Can you use Kafka and SQS together?");
System.out.println("A: Yes. Kafka for events, SQS for background jobs.");
System.out.println(" A Kafka consumer can enqueue SQS jobs.");
}
}
Expected output:
Q: Why not use Kafka for everything?
A: Kafka needs a cluster, partitions, consumer groups.
SQS is zero-ops for simple jobs.
Q: When does RabbitMQ beat Kafka?
A: Complex routing with exchanges and wildcard keys.
Kafka requires app-level filtering for the same.
Q: Can you use Kafka and SQS together?
A: Yes. Kafka for events, SQS for background jobs.
A Kafka consumer can enqueue SQS jobs.
Visual Overview
Kafka vs SQS vs RabbitMQ: Core Model ===================================== Kafka (Event Log): ┌────────────────────────────────────────────┐ │ transfer-events topic │ │ ┌──────────────────────────────────────┐ │ │ │ Partition 0: [e0][e1][e2][e3]... │ │ │ │ Partition 1: [e0][e1][e2]... │ │ │ │ Partition 2: [e0][e1][e2][e3][e4]... │ │ │ └──────────────────────────────────────┘ │ │ Consumer Group A ──reads──▶ (fraud) │ │ Consumer Group B ──reads──▶ (audit) │ │ Consumer Group C ──reads──▶ (notification) │ │ Events RETAINED after reading │ └────────────────────────────────────────────┘ SQS (Job Queue): ┌────────────────────────────────────────────┐ │ statement-queue │ │ ┌──────────────────────────────────────┐ │ │ │ [job1] [job2] [job3] [job4] │ │ │ └──────────┬───────────────────────────┘ │ │ │ │ │ Worker picks job1 │ │ Worker processes job1 │ │ Worker DELETES job1 │ │ job1 is GONE │ └────────────────────────────────────────────┘ RabbitMQ (Message Router): ┌────────────────────────────────────────────┐ │ Producer ──▶ loan-exchange (topic type) │ │ │ │ │ ┌──────────┼──────────┐ │ │ │ │ │ │ │ loan.home.* loan.*.mumbai loan.# │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ home-loan-q mumbai-q all-loans-q │ │ (home team) (mumbai) (analytics) │ └────────────────────────────────────────────┘
Key Takeaway
Choosing between Kafka, SQS, and RabbitMQ comes down to a simple mental model about what the message means. Kafka is for "something happened" scenarios: it is a durable, replayable event log where many independent consumer groups read the same events at their own pace. At National Bank of India, the transfer-events topic feeds the fraud service, the audit service, and the notification service independently, and the fraud team can replay last week's events whenever they need to retrain a model.
SQS is for "a job needs doing" scenarios: it is a fully managed cloud queue where one worker picks a job, does it, and deletes it, with zero infrastructure to manage. National Bank of India uses SQS for background tasks like generating monthly statement PDFs and sending email notifications. RabbitMQ is for "route this message" scenarios: it is a smart broker with exchanges, routing keys, and bindings that route messages to specific queues based on patterns. National Bank of India uses RabbitMQ to route loan applications to the right regional desk using topic exchange patterns like loan.home.mumbai.
The senior insight is that these are not competitors but complementary tools. A mature banking platform uses all three for the workloads each handles best: Kafka as the high-throughput event backbone with replay, SQS as the zero-ops job queue, and RabbitMQ as the routing-heavy broker. Choosing one system for everything is the most common mistake: Kafka is overkill for a simple email job, and SQS cannot replay events for a fraud model.
Remember one sentence: Kafka records what happened, SQS dispatches what needs doing, and RabbitMQ routes where a message should go.
---