Message Queue / Event Streaming: Kafka vs RabbitMQ vs SQS

• Message queue : an intermediary that decouples producers from consumers by passing messages asynchronously, enabling buffering, independent scaling, and resilience.
• RabbitMQ : a traditional message broker with exchanges, queues, and per-message acknowledgement, optimized for flexible routing and moderate throughput.
• Amazon SQS : a fully managed queue with at-least-once delivery (standard) or exactly-once (FIFO), requiring almost no operational burden.
• Apache Kafka : a distributed, partitioned, durable commit log optimized for high-throughput event streaming, replay, and multiple independent consumer groups.
• Dead-letter queue (DLQ) : a side queue that captures messages that fail processing after a retry limit, preventing a single poison message from blocking the main queue.

What is a message queue and why do we need it?

A message queue is an intermediary that sits between a producer and a consumer, accepting messages and delivering them asynchronously. The producer does not wait for the consumer to process the message, so a slow downstream system never blocks the fast upstream one. This decoupling also buffers bursts, because the queue absorbs a spike and the consumer drains it at its own pace.

package com.bank.messaging;

public class TransferEventPublisher {
    public void onTransferCompleted(Transfer t) {
        // Producer publishes and returns immediately
        kafkaTemplate.send("transfer.completed", t.accountId(), toJson(t));
        // Receipt, fraud, analytics happen later downstream
    }
}

How does RabbitMQ work?

RabbitMQ is a broker built around exchanges and queues. Producers publish to an exchange, which routes messages to queues based on rules (direct, topic, fanout), and consumers acknowledge each message so the broker knows it was handled. It excels at complex routing and per-message workflows, and traditionally deletes a message once acknowledged. It is a good fit when you need rich routing and moderate throughput.

package com.bank.messaging;

public class RabbitMqExample {
    // Direct exchange routes by routing key
    public void publishTransfer(Transfer t) {
        rabbitTemplate.convertAndSend(
            "banking.exchange", "transfer.completed", t);
    }
    // Consumer acknowledges after processing
    @RabbitListener(queues = "receipt.queue")
    public void handleReceipt(Transfer t) {
        receiptService.send(t);
        // auto-ack on success; nack on exception triggers retry
    }
}

How does Amazon SQS work?

Amazon SQS is a fully managed queue with almost no operational burden. A standard queue gives at-least-once delivery and best-effort ordering with massive scale, while a FIFO queue gives strict ordering and exactly-once processing within throughput limits. It is ideal when you want a simple, reliable queue without running infrastructure.

package com.bank.messaging;

import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.*;

public class SqsExample {
    public void send(SqsClient sqs, String queueUrl, String body) {
        sqs.sendMessage(SendMessageRequest.builder()
            .queueUrl(queueUrl).messageBody(body).build());
    }
    public void receive(SqsClient sqs, String queueUrl) {
        var msgs = sqs.receiveMessage(ReceiveMessageRequest.builder()
            .queueUrl(queueUrl).maxNumberOfMessages(10).build()).messages();
        for (var m : msgs) {
            process(m.body());
            sqs.deleteMessage(DeleteMessageRequest.builder()
                .queueUrl(queueUrl).receiptHandle(m.receiptHandle()).build());
        }
    }
}

How does Apache Kafka work?

Kafka is a distributed commit log. Messages are appended to topics split into partitions, retained for a configurable time regardless of consumption, so consumers can replay history. It is built for very high throughput and for multiple independent consumer groups reading the same stream. Partitioning by key keeps related messages ordered on the same partition.

package com.bank.messaging;

import org.apache.kafka.clients.producer.ProducerRecord;

public class KafkaExample {
    // Produce to a partition by key so same-account messages stay ordered
    public void publish(String accountId, String transferJson) {
        producer.send(new ProducerRecord<>("transfers", accountId, transferJson));
    }
    // Consumer group: each partition read by exactly one consumer in the group
    public void consume() {
        consumer.subscribe(List.of("transfers"));
        while (true) {
            var records = consumer.poll(Duration.ofMillis(100));
            records.forEach(r -> process(r.value()));
        }
    }
}

What are the delivery semantics?

Delivery semantics matter enormously in banking. At-most-once means a message may be lost but never duplicated. At-least-once means a message may be redelivered, so consumers must be idempotent. Exactly-once is harder and, in Kafka, achieved with transactional producers and idempotent writes within the Kafka boundary. In practice I design for at-least-once plus idempotent consumers, because that is simpler and robust.

package com.bank.messaging;

public class IdempotentConsumer {
    public void handle(TransferEvent event) {
        if (processedIds.contains(event.eventId())) return; // duplicate, skip
        receiptService.send(event);
        processedIds.add(event.eventId());
    }
}

What are consumer groups?

Consumer groups give horizontal scaling. A group shares a topic's partitions among its members so each message is processed once per group, and adding members increases parallelism up to the partition count. Multiple groups can read the same topic independently : the receipt group and the fraud group each see every message.

package com.bank.messaging;

public class ConsumerGroupExample {
    // Group "receipt-group" : 3 consumers share 6 partitions (2 each)
    // Group "fraud-group"   : 2 consumers share 6 partitions (3 each)
    // Both groups see every message independently
}

What is a dead-letter queue?

A dead-letter queue captures messages that repeatedly fail processing. After a configurable retry limit, the message is moved aside for investigation rather than retrying forever or being silently lost. This prevents a single poison message from blocking the entire queue and gives operations a place to examine and fix failures.

package com.bank.messaging;

public class DeadLetterHandler {
    public void handleWithRetry(Message msg) {
        int attempts = msg.retryCount();
        if (attempts >= MAX_RETRIES) {
            deadLetterQueue.send(msg);  // move aside for investigation
            return;
        }
        try { process(msg); }
        catch (Exception e) { retryQueue.sendWithDelay(msg, backoff(attempts)); }
    }
}

How do you choose between them?

Choose RabbitMQ when you need rich routing patterns, moderate throughput, and per-message acknowledgement. Choose SQS when you want a managed queue with zero operational overhead and your workload is simple point-to-point. Choose Kafka when you need high-throughput streaming, message replay, event sourcing, or multiple independent consumer groups reading the same stream. In banking, Kafka is the typical choice for the event backbone because it retains events for replay and supports many consumer groups.

What should a senior engineer know?

Ordering guarantees are per partition, so if you need strict ordering for an account, you partition by account id, accepting that this limits parallelism for a hot account. A senior engineer also designs for backpressure : when consumers fall behind, the queue grows, and the system must either scale consumers, shed load, or alert, rather than silently accumulating unbounded lag.