Definition
Kafka and RabbitMQ are both messaging systems but with fundamentally different models.
- Kafka – a distributed, durable, append-only commit log where messages are retained on disk for a configured time and consumers track their own offset, enabling replay and multiple independent consumer groups.
- RabbitMQ – a traditional message broker implementing AMQP with exchanges, routing keys, bindings, and queues, where a consumed and acknowledged message is removed from the queue.
- Retention – Kafka keeps messages regardless of consumption; RabbitMQ deletes messages after acknowledgement.
- Routing – Kafka uses flat topics with consumer-side filtering; RabbitMQ provides broker-side routing via direct, topic, fanout, and headers exchanges.
- Consumption model – Kafka consumer groups each read the full stream; RabbitMQ competing consumers share a queue so each message is processed once.
At National Bank of India, Kafka is the permanent transaction ledger. Every event is written in order and kept for years, and any department can read from wherever they left off or even re-read last week's entries. RabbitMQ is the internal mail sorting room with clever routing rules. A letter is dropped at the sorting desk, the desk routes it to the right pigeonhole based on the address label, and once a clerk picks it up and signs for it, the letter is gone. The ledger is for keeping and replaying events; the mail room is for delivering each task to the right worker once.
Understanding the Concepts
What is Kafka's core model?
Kafka is a distributed, append-only commit log. Producers append records to a topic partition, and the records are stored on disk for the configured retention period regardless of whether anyone reads them. Each consumer group tracks its own offset, which is simply a position in the log.
This design means Kafka supports replay: a consumer can reset its offset and re-read retained records. It also means multiple independent consumer groups each get the complete stream without interfering with each other.
package com.bank.kafka;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.math.BigDecimal;
import java.util.Properties;
/**
* Rajesh publishes a transaction to the durable Kafka log.
* The record stays on disk for 7 days regardless of who reads it.
*/
public class KafkaTransactionProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-1:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
String accountId = "SAV2001";
BigDecimal amount = new BigDecimal("75000.00");
String value = "{\"account\":\"" + accountId
+ "\",\"amount\":" + amount
+ ",\"type\":\"CREDIT\"}";
producer.send(new ProducerRecord<>("bank.transactions", accountId, value));
System.out.println("Published to Kafka log. Retained for 7 days.");
System.out.println("Analytics, fraud, and ledger teams each read independently.");
}
}
}
Expected output:
Published to Kafka log. Retained for 7 days.
Analytics, fraud, and ledger teams each read independently.
What is RabbitMQ's core model?
RabbitMQ implements the AMQP exchange-and-queue model. A producer publishes to an exchange, not directly to a queue. The exchange routes the message into one or more queues according to bindings and the message's routing key. A consumer reads from a queue, and when it acknowledges the message, RabbitMQ removes it.
RabbitMQ has four exchange types:
- Direct exchange matches the routing key exactly.
- Topic exchange matches wildcard patterns (e.g.,
loan.#matchesloan.personal.approval). - Fanout exchange broadcasts to all bound queues.
- Headers exchange matches on message header attributes.
package com.bank.rabbitmq;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import java.math.BigDecimal;
/**
* Rajesh publishes a loan application to RabbitMQ.
* The exchange routes it to the correct processing queue.
*/
public class RabbitMqLoanPublisher {
public static void main(String[] args) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("rabbitmq-1");
try (Connection connection = factory.newConnection();
Channel channel = connection.createChannel()) {
// Declare a topic exchange for loan routing
channel.exchangeDeclare("loan.exchange", "topic", true);
String routingKey = "loan.personal.urgent";
BigDecimal loanAmount = new BigDecimal("500000.00");
String message = "{\"applicant\":\"Priya\","
+ "\"account\":\"SAV2001\","
+ "\"amount\":" + loanAmount + "}";
channel.basicPublish("loan.exchange", routingKey, null, message.getBytes());
System.out.println("Published with routing key: " + routingKey);
System.out.println("Exchange routes to matching queue. Once acked, message is gone.");
}
}
}
Expected output:
Published with routing key: loan.personal.urgent
Exchange routes to matching queue. Once acked, message is gone.
How is Kafka different from RabbitMQ?
The two systems differ in their core model, retention, routing, consumption, and features.
| Feature | Kafka | RabbitMQ |
|---|---|---|
| Core model | Append-only distributed log | Exchange-queue message broker (AMQP) |
| Retention | Messages kept for configured time (days/weeks) | Messages deleted after acknowledgement |
| Replay | Yes, reset consumer offset to any point | No built-in replay |
| Routing | Flat topics, consumer-side filtering | Broker-side routing via exchanges and routing keys |
| Consumption | Consumer groups each get the full stream | Competing consumers share a queue (each message once) |
| Ordering | Per-partition ordering guaranteed | Per-queue ordering (single consumer) |
| Throughput | Very high (sequential disk I/O, batching) | Lower throughput, lower per-message latency |
| Message priority | Not supported natively | Built-in priority queues |
| Request-reply (RPC) | Not native, requires manual correlation | Built-in reply-to queue pattern |
| TTL per message | Not supported (topic-level retention) | Per-message TTL supported |
| Protocol | Kafka binary protocol | AMQP, MQTT, STOMP |
package com.bank.comparison;
public class KafkaVsRabbitMq {
public static void main(String[] args) {
System.out.println("=== Kafka ===");
System.out.println("Model: append-only log");
System.out.println("Retention: 7 days (configurable)");
System.out.println("Replay: yes (reset offset)");
System.out.println("Routing: flat topics, filter in consumer");
System.out.println("Throughput: millions of records/sec");
System.out.println();
System.out.println("=== RabbitMQ ===");
System.out.println("Model: exchange + queue broker");
System.out.println("Retention: message removed after ack");
System.out.println("Replay: no (message is gone)");
System.out.println("Routing: broker-side, exchange types");
System.out.println("Latency: lower per-message latency");
}
}
Expected output:
=== Kafka ===
Model: append-only log
Retention: 7 days (configurable)
Replay: yes (reset offset)
Routing: flat topics, filter in consumer
Throughput: millions of records/sec
=== RabbitMQ ===
Model: exchange + queue broker
Retention: message removed after ack
Replay: no (message is gone)
Routing: broker-side, exchange types
Latency: lower per-message latency
How do consumption models differ?
In Kafka, consumers form consumer groups. Each consumer group gets the full stream because each group tracks its own offset independently. Within a group, partitions are divided among consumers so each record is processed once per group. This makes Kafka ideal for fan-out to multiple independent systems.
In RabbitMQ, consumers compete on a queue. A message is delivered to one consumer among those attached to the queue, and once acknowledged it is removed. This is the natural task-queue pattern where each job is done once by one worker. Broadcasting to many consumers requires a fanout exchange binding separate queues.
package com.bank.comparison;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
/**
* Kafka: three independent consumer groups each get the FULL stream.
* Analytics, Fraud, and Ledger teams all read every transaction.
*/
public class KafkaConsumerGroupDemo {
public static void main(String[] args) {
// Each team creates its own consumer group
String[] teams = {"analytics-group", "fraud-group", "ledger-group"};
for (String groupId : teams) {
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-1:9092");
props.put("group.id", groupId);
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("auto.offset.reset", "earliest");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Collections.singletonList("bank.transactions"));
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
System.out.println(groupId + " received " + records.count() + " records (full stream)");
}
}
}
}
Expected output:
analytics-group received 1000 records (full stream)
fraud-group received 1000 records (full stream)
ledger-group received 1000 records (full stream)
When should you choose Kafka?
Choose Kafka when the dominant requirement is durable streaming, replay, or high throughput.
- Event sourcing where the log of events is the source of truth and you rebuild state by replaying.
- Log aggregation where application logs from hundreds of services are collected into one stream.
- Stream processing using Kafka Streams or ksqlDB to compute aggregations, joins, and windows.
- Multiple independent consumers where analytics, fraud, compliance, and ledger each need the full stream.
- Long-term retention where events must be kept for days, weeks, or years.
At National Bank of India, Pankaj chose Kafka as the event backbone because the analytics, fraud, and ledger teams all needed the complete transaction stream independently, and compliance needed seven-year retention with the ability to replay.
# Replay: reset the ledger consumer group to re-read all events
bin/kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 \
--group ledger-group \
--topic bank.transactions \
--reset-offsets --to-earliest --execute
Expected output:
GROUP TOPIC PARTITION NEW-OFFSET
ledger-group bank.transactions 0 0
ledger-group bank.transactions 1 0
ledger-group bank.transactions 2 0
When should you choose RabbitMQ?
Choose RabbitMQ when the dominant requirement is task routing, request-reply, or per-message features.
- Task queues where each job (loan application, payment processing) is handled once by one worker.
- Request-reply RPC where a service sends a request and waits for a response on a reply-to queue.
- Complex broker-side routing where exchanges route messages by type, priority, or header attributes.
- Per-message priority where urgent loan applications jump ahead of normal ones.
- Per-message TTL where a quote request expires if not processed within 30 seconds.
At National Bank of India, Rajesh chose RabbitMQ for the loan processing pipeline because each application must be processed exactly once by one worker, urgent applications need priority, and the branch portal uses request-reply to get instant loan eligibility quotes.
package com.bank.rabbitmq;
import com.rabbitmq.client.*;
import java.math.BigDecimal;
/**
* RabbitMQ RPC: branch portal sends a loan eligibility request
* and waits for a reply. Kafka does not support this natively.
*/
public class LoanEligibilityRpc {
public static void main(String[] args) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("rabbitmq-1");
try (Connection connection = factory.newConnection();
Channel channel = connection.createChannel()) {
// Create exclusive reply queue
String replyQueue = channel.queueDeclare().getQueue();
String correlationId = java.util.UUID.randomUUID().toString();
AMQP.BasicProperties props = new AMQP.BasicProperties.Builder()
.correlationId(correlationId)
.replyTo(replyQueue)
.build();
String request = "{\"applicant\":\"Priya\","
+ "\"account\":\"SAV2001\","
+ "\"requestedAmount\":" + new BigDecimal("500000.00") + "}";
channel.basicPublish("", "loan.eligibility.rpc", props, request.getBytes());
System.out.println("Sent eligibility request for Priya (SAV2001)");
// Wait for reply
GetResponse response = channel.basicGet(replyQueue, true);
if (response != null) {
System.out.println("Reply: " + new String(response.getBody()));
}
}
}
}
Expected output:
Sent eligibility request for Priya (SAV2001)
Reply: {"eligible":true,"maxAmount":750000.00,"interestRate":8.5}
What are common mistakes when choosing between them?
The most common mistake is forcing one system to do the other's job.
Wrong: using Kafka as a task queue with per-message priority.
Kafka does not support per-message priority. Within a partition, messages are strictly ordered by offset. You cannot make one message jump ahead of another.
package com.bank.comparison;
public class WrongKafkaAsTaskQueue {
public static void main(String[] args) {
System.out.println("WRONG: Using Kafka for priority task queue");
System.out.println("Kafka partition order: [msg1, msg2-URGENT, msg3]");
System.out.println("msg2-URGENT cannot jump ahead of msg1");
System.out.println("Kafka has no per-message priority.");
System.out.println();
System.out.println("CORRECT: Use RabbitMQ with priority queue");
System.out.println("RabbitMQ queue: [msg1(priority=1), msg2(priority=10), msg3(priority=1)]");
System.out.println("msg2 with priority=10 is delivered first.");
}
}
Expected output:
WRONG: Using Kafka for priority task queue
Kafka partition order: [msg1, msg2-URGENT, msg3]
msg2-URGENT cannot jump ahead of msg1
Kafka has no per-message priority.
CORRECT: Use RabbitMQ with priority queue
RabbitMQ queue: [msg1(priority=1), msg2(priority=10), msg3(priority=1)]
msg2 with priority=10 is delivered first.
Wrong: using RabbitMQ for event sourcing with replay.
RabbitMQ removes messages after acknowledgement. You cannot reset to the beginning and re-read all events. RabbitMQ Streams (a newer feature) add log-like retention, but the classic queue model does not support replay.
package com.bank.comparison;
public class WrongRabbitAsEventStore {
public static void main(String[] args) {
System.out.println("WRONG: Using RabbitMQ for event sourcing");
System.out.println("Messages are deleted after ack.");
System.out.println("Cannot replay 6 months of transactions.");
System.out.println();
System.out.println("CORRECT: Use Kafka with long retention");
System.out.println("retention.ms=15552000000 (180 days)");
System.out.println("Reset offset to earliest and replay all events.");
}
}
Expected output:
WRONG: Using RabbitMQ for event sourcing
Messages are deleted after ack.
Cannot replay 6 months of transactions.
CORRECT: Use Kafka with long retention
retention.ms=15552000000 (180 days)
Reset offset to earliest and replay all events.
What about using both together?
Many banks run Kafka and RabbitMQ side by side, each handling what it does best. Kafka serves as the durable event backbone for streaming, replay, and analytics. RabbitMQ handles task routing, RPC, and priority processing.
At National Bank of India, the transaction stream flows through Kafka where analytics, fraud, and the ledger each read independently. Loan applications are routed through RabbitMQ where each application goes to one worker, urgent ones get priority, and the branch portal uses RPC for instant eligibility checks.
package com.bank.comparison;
public class BothTogether {
public static void main(String[] args) {
System.out.println("=== National Bank of India Architecture ===");
System.out.println();
System.out.println("Kafka (event backbone):");
System.out.println(" - bank.transactions topic (7-day retention)");
System.out.println(" - Analytics team: reads full stream");
System.out.println(" - Fraud team: reads full stream");
System.out.println(" - Ledger team: reads full stream + replay");
System.out.println();
System.out.println("RabbitMQ (task routing):");
System.out.println(" - loan.exchange (topic exchange)");
System.out.println(" - loan.personal.urgent -> priority queue");
System.out.println(" - loan.personal.normal -> standard queue");
System.out.println(" - loan.eligibility.rpc -> request-reply");
}
}
Expected output:
=== National Bank of India Architecture ===
Kafka (event backbone):
- bank.transactions topic (7-day retention)
- Analytics team: reads full stream
- Fraud team: reads full stream
- Ledger team: reads full stream + replay
RabbitMQ (task routing):
- loan.exchange (topic exchange)
- loan.personal.urgent -> priority queue
- loan.personal.normal -> standard queue
- loan.eligibility.rpc -> request-reply
What should a senior engineer know?
A senior engineer understands the nuances beyond the basic comparison.
- Kafka can emulate a work queue using a consumer group with one partition per worker, but it lacks per-message priority and fine-grained routing.
- RabbitMQ can emulate pub-sub with a fanout exchange, but it is not built for long retention, replay, or log-level throughput.
- RabbitMQ Streams (introduced in 3.9) add log-like retention with offset tracking, bringing some Kafka-like capabilities, but the classic queue model remains the primary abstraction.
- Kafka Streams and ksqlDB give Kafka a native stream-processing ecosystem that RabbitMQ does not have.
- In a bank, Kafka's retention lets compliance replay events for audits, which is impossible with RabbitMQ's delete-on-ack model.
package com.bank.comparison;
public class SeniorNuances {
public static void main(String[] args) {
System.out.println("Senior-level comparison:");
System.out.println("1. Kafka consumer group ~ work queue (but no priority)");
System.out.println("2. RabbitMQ fanout ~ pub-sub (but no replay)");
System.out.println("3. RabbitMQ Streams add log-like retention (3.9+)");
System.out.println("4. Kafka Streams/ksqlDB = native stream processing");
System.out.println("5. Kafka retention = compliance replay in banking");
System.out.println("6. Both together is the common bank pattern");
}
}
Expected output:
Senior-level comparison:
1. Kafka consumer group ~ work queue (but no priority)
2. RabbitMQ fanout ~ pub-sub (but no replay)
3. RabbitMQ Streams add log-like retention (3.9+)
4. Kafka Streams/ksqlDB = native stream processing
5. Kafka retention = compliance replay in banking
6. Both together is the common bank pattern
Visual Overview
Kafka Model (Append-Only Log):
Producer ──► Topic Partition [msg1][msg2][msg3][msg4][msg5]
▲ ▲
│ │
offset=0 offset=4
(replay from (current
beginning) position)
Consumer Group A (analytics): reads ALL from offset 0
Consumer Group B (fraud): reads ALL from offset 0
Consumer Group C (ledger): reads ALL from offset 0
Messages STAY on disk for retention period.
─────────────────────────────────────────────────
RabbitMQ Model (Exchange + Queue):
Producer ──► Exchange ──routing──► Queue ──► Consumer
│ │
│ direct/topic/ │ message acked
│ fanout/headers │ = REMOVED
│ │
└──routing──► Queue2 ──► Consumer2
Each message delivered to ONE consumer per queue.
After ack, message is GONE from queue.
─────────────────────────────────────────────────
National Bank of India:
┌──────────────────────┐ ┌──────────────────────┐
│ Kafka │ │ RabbitMQ │
│ (event backbone) │ │ (task routing) │
│ │ │ │
│ bank.transactions │ │ loan.exchange │
│ ├─ analytics-group │ │ ├─ urgent queue │
│ ├─ fraud-group │ │ ├─ normal queue │
│ └─ ledger-group │ │ └─ rpc reply queue │
└──────────────────────┘ └──────────────────────┘
Key Takeaway
Kafka and RabbitMQ are both messaging systems, but they are built on fundamentally different models. Kafka is a distributed, durable, append-only log where messages are retained on disk for a configured period and each consumer group tracks its own offset. This means Kafka supports replay, multiple independent consumers of the same stream, and very high throughput from sequential disk I/O. RabbitMQ is a traditional AMQP broker where producers publish to exchanges that route messages by binding rules and routing keys into queues, and once a consumer acknowledges a message it is removed.
The choice follows the workload. At National Bank of India, Pankaj chose Kafka as the event backbone because the analytics, fraud, and ledger teams all needed the complete transaction stream independently, and compliance needed seven-year retention with replay capability. Rajesh chose RabbitMQ for the loan processing pipeline because each application must be handled once by one worker, urgent applications need priority queues, and the branch portal uses request-reply RPC for instant eligibility checks.
The senior insight is that each system can poorly imitate the other, but forcing one into the other's role creates problems. Kafka can emulate a work queue but lacks per-message priority and fine routing. RabbitMQ can emulate pub-sub with fanout but is not built for long retention, replay, or log-level throughput. Many banks run both side by side, each handling what its model does best.
Remember one sentence: Kafka keeps every event in a durable log for replay and fan-out; RabbitMQ routes each task to the right worker once and then forgets it.
---