Back to Kafka Fundamentals: Topics, Partitions, Offsets
Q19Question 19 of 19

What are message brokers and how do queues and topics work

Answer: A message broker is middleware that sits between services, accepts messages from producers, stores them temporarily, and delivers them to consumers so the two sides do not need to communicate directly or at the same time.

Key Concepts
  • A message broker is middleware that accepts messages from producers and delivers them to consumers without requiring both to be online at the same time.
  • A queue delivers each message to exactly one consumer, which is ideal for task distribution like loan application processing.
  • A topic delivers each message to all subscribers, which is ideal for event broadcasting like account balance changes.
  • Kafka uses a distributed commit log with partitions and consumer groups to combine queue and topic semantics.
  • Asynchronous communication decouples services so a failure in one does not block the others.
  • Exactly-once delivery requires idempotent producers and transactional consumers working together.
  • Choosing between queue and topic depends on whether the message represents a task (one handler) or an event (many listeners).

Definition

A message broker is middleware that sits between services, accepts messages from producers, stores them temporarily, and delivers them to consumers so the two sides do not need to communicate directly or at the same time.
  • Message broker is a server that receives messages from producer applications and routes them to consumer applications using queues or topics.
  • Queue is a channel where each message is delivered to exactly one consumer, following first-in-first-out (FIFO) order.
  • Topic is a channel where each message is delivered to every active subscriber, enabling fan-out communication.
  • Subscription is a consumer's registration on a topic, telling the broker to deliver a copy of every published message.
  • Asynchronous communication means the producer sends a message and continues working without waiting for the consumer to process it.

Think of the National Bank of India branch office. When a customer submits a loan application at the counter, the clerk does not process it immediately. The clerk drops the application into a tray (the queue). A loan officer picks it up later and processes it. The customer does not wait at the counter. Now imagine the bank manager announces a new interest rate. That announcement goes on the notice board (the topic). Every department (home loans, car loans, personal loans) reads the same announcement independently. The tray is a queue: one person picks up each form. The notice board is a topic: everyone reads every notice.

Understanding the Concepts

What is a message broker?

A message broker is a dedicated server that sits between producers and consumers. The producer sends a message to the broker. The broker stores the message and delivers it to the right consumer when the consumer is ready.

Without a broker, Service A must call Service B directly over HTTP. If Service B is down, the call fails. With a broker in between, Service A drops the message into the broker and moves on. Service B picks it up when it comes back online.

The three main responsibilities of a message broker are:

  • Decoupling. The producer does not need to know who the consumer is or where it runs.
  • Buffering. Messages are stored until the consumer is ready, absorbing traffic spikes.
  • Routing. The broker decides which consumer gets which message based on queues, topics, or routing rules.
JAVA
package com.bank.messaging;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

public class SimpleBrokerDemo {

    public static void main(String[] args) throws InterruptedException {
        // A blocking queue simulates a simple message broker
        BlockingQueue<String> loanQueue = new LinkedBlockingQueue<>();

        // Producer: Rajesh submits a loan application
        loanQueue.put("LOAN_APP:Rajesh:SAV2001:500000");
        System.out.println("Producer: Submitted loan application for Rajesh");

        // Consumer: Loan officer picks it up later
        String message = loanQueue.take();
        System.out.println("Consumer: Processing " + message);
    }
}
TEXT
Producer: Submitted loan application for Rajesh
Consumer: Processing LOAN_APP:Rajesh:SAV2001:500000

How does a queue work?

A queue follows point-to-point messaging. Each message goes to exactly one consumer. If three consumers listen on the same queue, the broker distributes messages round-robin so that each message is processed only once.

Queues are ideal when the message represents a task that should be done exactly once: process a loan, send an OTP, generate a statement.

  • Messages are stored in FIFO order.
  • Once a consumer acknowledges a message, it is removed from the queue.
  • If the consumer crashes before acknowledging, the broker redelivers the message to another consumer.
JAVA
package com.bank.messaging;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

public class QueueDemo {

    public static void main(String[] args) throws InterruptedException {
        BlockingQueue<String> transferQueue = new LinkedBlockingQueue<>();

        // Three transfer requests from different customers
        transferQueue.put("TXN:Rajesh:SAV2001->SAV3002:10000");
        transferQueue.put("TXN:Priya:SAV3002->SAV4003:25000");
        transferQueue.put("TXN:Amit:SAV5005->SAV2001:15000");

        // Two consumers compete for messages
        // Each message goes to exactly one consumer
        System.out.println("Consumer-1 processes: " + transferQueue.take());
        System.out.println("Consumer-2 processes: " + transferQueue.take());
        System.out.println("Consumer-1 processes: " + transferQueue.take());
    }
}
TEXT
Consumer-1 processes: TXN:Rajesh:SAV2001->SAV3002:10000
Consumer-2 processes: TXN:Priya:SAV3002->SAV4003:25000
Consumer-1 processes: TXN:Amit:SAV5005->SAV2001:15000

How does a topic work?

A topic follows publish-subscribe messaging. Every subscriber receives a copy of every message. If three services subscribe to the "account-events" topic, all three get every account event.

Topics are ideal when the message represents an event that multiple services care about: account balance changed, loan approved, fraud detected.

  • The producer publishes once.
  • Each subscriber gets its own copy.
  • Subscribers can join and leave independently without affecting each other.
JAVA
package com.bank.messaging;

import java.util.List;
import java.util.ArrayList;
import java.util.function.Consumer;

public class TopicDemo {

    private final List<Consumer<String>> subscribers = new ArrayList<>();

    public void subscribe(String name, Consumer<String> handler) {
        subscribers.add(msg -> {
            System.out.println(name + " received: " + msg);
            handler.accept(msg);
        });
    }

    public void publish(String message) {
        System.out.println("Published: " + message);
        for (Consumer<String> sub : subscribers) {
            sub.accept(message);
        }
    }

    public static void main(String[] args) {
        TopicDemo accountTopic = new TopicDemo();

        // Three services subscribe to account events
        accountTopic.subscribe("NotificationService", msg -> {});
        accountTopic.subscribe("FraudDetectionService", msg -> {});
        accountTopic.subscribe("StatementService", msg -> {});

        // One event published, all three receive it
        accountTopic.publish("BALANCE_CHANGED:SAV2001:Rajesh:75000");
    }
}
TEXT
Published: BALANCE_CHANGED:SAV2001:Rajesh:75000
NotificationService received: BALANCE_CHANGED:SAV2001:Rajesh:75000
FraudDetectionService received: BALANCE_CHANGED:SAV2001:Rajesh:75000
StatementService received: BALANCE_CHANGED:SAV2001:Rajesh:75000

How is a queue different from a topic?

FeatureQueueTopic
DeliveryEach message to one consumerEach message to all subscribers
Use caseTask processing (process loan, send OTP)Event broadcasting (balance changed, fraud alert)
Consumer countCompeting consumers share the loadEach subscriber gets every message
Message removalRemoved after acknowledgmentRetained for a configured retention period (Kafka)
ReplayNot possible after consumptionPossible by resetting offset (Kafka)
ScalingAdd consumers to process fasterAdd subscribers for new features
ExampleJMS Queue, SQS, RabbitMQ queueKafka topic, SNS, RabbitMQ exchange+fanout
JAVA
package com.bank.messaging;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.List;
import java.util.ArrayList;
import java.util.function.Consumer;

public class QueueVsTopicDemo {

    public static void main(String[] args) throws InterruptedException {
        System.out.println("=== Queue: Each message to one consumer ===");
        BlockingQueue<String> loanQueue = new LinkedBlockingQueue<>();
        loanQueue.put("LOAN:Rajesh:500000");
        loanQueue.put("LOAN:Priya:300000");
        // Only one consumer gets each message
        System.out.println("Officer-A: " + loanQueue.take());
        System.out.println("Officer-B: " + loanQueue.take());

        System.out.println();
        System.out.println("=== Topic: Each message to all subscribers ===");
        List<Consumer<String>> subs = new ArrayList<>();
        subs.add(msg -> System.out.println("SMS-Service: " + msg));
        subs.add(msg -> System.out.println("Email-Service: " + msg));
        subs.add(msg -> System.out.println("Audit-Service: " + msg));

        String event = "FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer";
        for (Consumer<String> sub : subs) {
            sub.accept(event);
        }
    }
}
TEXT
=== Queue: Each message to one consumer ===
Officer-A: LOAN:Rajesh:500000
Officer-B: LOAN:Priya:300000

=== Topic: Each message to all subscribers ===
SMS-Service: FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer
Email-Service: FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer
Audit-Service: FRAUD_ALERT:SAV2001:Rajesh:suspicious_transfer

How does Kafka combine queue and topic semantics?

Kafka uses a distributed commit log model that is different from traditional brokers. A Kafka topic is a log divided into partitions. Each partition is an ordered, append-only sequence of messages.

Kafka achieves queue behavior through consumer groups: all consumers in the same group share the partitions, so each message goes to one consumer in the group. Kafka achieves topic behavior by having multiple consumer groups: each group independently reads every message.

Key Kafka concepts:

  • Topic. A named category of messages. Example: account-transactions.
  • Partition. A segment of a topic. Partitions allow parallel processing.
  • Offset. A sequential number assigned to each message in a partition.
  • Consumer group. A set of consumers that cooperate to read from a topic, where each partition is assigned to exactly one consumer in the group.
  • Broker. A Kafka server that stores partitions and serves read/write requests.
JAVA
package com.bank.messaging;

import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;

public class KafkaTransactionProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
                  StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
                  StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            // Key = account ID ensures all transactions for one account
            // go to the same partition (ordering guarantee)
            producer.send(new ProducerRecord<>(
                "account-transactions", "SAV2001",
                "{\"from\":\"SAV2001\",\"to\":\"SAV3002\",\"amount\":10000}"
            ));
            System.out.println("Sent transaction event for SAV2001");

            producer.send(new ProducerRecord<>(
                "account-transactions", "SAV3002",
                "{\"from\":\"SAV3002\",\"to\":\"SAV4003\",\"amount\":25000}"
            ));
            System.out.println("Sent transaction event for SAV3002");
        }
    }
}
TEXT
Sent transaction event for SAV2001
Sent transaction event for SAV3002
JAVA
package com.bank.messaging;

import java.time.Duration;
import java.util.List;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class KafkaTransactionConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "transaction-processor");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
                  StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
                  StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("account-transactions"));

            ConsumerRecords<String, String> records =
                consumer.poll(Duration.ofSeconds(5));

            records.forEach(record ->
                System.out.printf("Partition=%d Offset=%d Key=%s Value=%s%n",
                    record.partition(), record.offset(),
                    record.key(), record.value())
            );
        }
    }
}
TEXT
Partition=0 Offset=0 Key=SAV2001 Value={"from":"SAV2001","to":"SAV3002","amount":10000}
Partition=1 Offset=0 Key=SAV3002 Value={"from":"SAV3002","to":"SAV4003","amount":25000}

What are common mistakes with message brokers?

Mistake 1: Not handling duplicate messages

Wrong: assuming each message arrives exactly once.

JAVA
package com.bank.messaging;

import java.math.BigDecimal;

public class NonIdempotentConsumer {

    private BigDecimal balance = new BigDecimal("100000.00");

    // WRONG: If the broker redelivers the message, the amount
    // is credited twice. The customer gets double the money.
    public void handleDeposit(String transactionId, BigDecimal amount) {
        balance = balance.add(amount);
        System.out.println("Balance after deposit: " + balance);
    }

    public static void main(String[] args) {
        NonIdempotentConsumer consumer = new NonIdempotentConsumer();
        // Same message delivered twice (broker retry after timeout)
        consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
        consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
    }
}
TEXT
Balance after deposit: 150000.00
Balance after deposit: 200000.00

Correct: use an idempotency key to detect and skip duplicate messages.

JAVA
package com.bank.messaging;

import java.math.BigDecimal;
import java.util.Set;
import java.util.HashSet;

public class IdempotentConsumer {

    private BigDecimal balance = new BigDecimal("100000.00");
    private final Set<String> processedIds = new HashSet<>();

    // CORRECT: Check if we already processed this transaction.
    // If yes, skip it. The balance stays correct.
    public void handleDeposit(String transactionId, BigDecimal amount) {
        if (processedIds.contains(transactionId)) {
            System.out.println("Duplicate skipped: " + transactionId);
            return;
        }
        processedIds.add(transactionId);
        balance = balance.add(amount);
        System.out.println("Balance after deposit: " + balance);
    }

    public static void main(String[] args) {
        IdempotentConsumer consumer = new IdempotentConsumer();
        consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
        consumer.handleDeposit("TXN-1001", new BigDecimal("50000.00"));
    }
}
TEXT
Balance after deposit: 150000.00
Duplicate skipped: TXN-1001

Mistake 2: Not setting a dead letter queue

Wrong: letting a poison message (one that always fails) block the entire queue forever.

Correct: after a configured number of retries, move the message to a dead letter queue for manual inspection.

JAVA
package com.bank.messaging;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

public class DeadLetterQueueDemo {

    static final int MAX_RETRIES = 3;

    public static void main(String[] args) throws InterruptedException {
        BlockingQueue<String> mainQueue = new LinkedBlockingQueue<>();
        BlockingQueue<String> deadLetterQueue = new LinkedBlockingQueue<>();

        mainQueue.put("TRANSFER:SAV2001:INVALID_ACCOUNT:-5000");

        String message = mainQueue.take();
        int retries = 0;
        boolean processed = false;

        while (retries < MAX_RETRIES && !processed) {
            try {
                processTransfer(message);
                processed = true;
            } catch (Exception e) {
                retries++;
                System.out.println("Retry " + retries + ": " + e.getMessage());
            }
        }

        if (!processed) {
            deadLetterQueue.put(message);
            System.out.println("Moved to DLQ: " + message);
        }
    }

    static void processTransfer(String msg) {
        if (msg.contains("INVALID_ACCOUNT")) {
            throw new IllegalArgumentException("Invalid target account");
        }
    }
}
TEXT
Retry 1: Invalid target account
Retry 2: Invalid target account
Retry 3: Invalid target account
Moved to DLQ: TRANSFER:SAV2001:INVALID_ACCOUNT:-5000

How do you choose between synchronous and asynchronous communication?

CriteriaSynchronous (HTTP/gRPC)Asynchronous (Broker)
Response needed immediatelyYes: check balance, authenticateNo: send email, generate PDF
CouplingTight: caller waits for responseLoose: fire and forget
Failure handlingCaller retries or failsBroker retries automatically
Throughput spikesCaller overwhelms the calleeBroker buffers the spike
Ordering guaranteePer-request orderingPartition-level ordering (Kafka)
DebuggingEasier: single request/responseHarder: distributed trace needed

In the National Bank of India system:

  • Synchronous. Rajesh checks his account balance. The mobile app calls the Account Service over HTTP and waits for the response.
  • Asynchronous. Priya transfers money. The Transfer Service publishes a TransferInitiated event to Kafka. The Notification Service, Statement Service, and Fraud Detection Service each consume the event independently.
JAVA
package com.bank.service;

import java.math.BigDecimal;

public class CommunicationChoiceDemo {

    // Synchronous: caller waits for the result
    static BigDecimal checkBalance(String accountId) {
        System.out.println("[SYNC] Fetching balance for " + accountId);
        return new BigDecimal("75000.00");
    }

    // Asynchronous: producer fires and forgets
    static void initiateTransfer(String from, String to, BigDecimal amount) {
        String event = String.format(
            "TransferInitiated{from=%s, to=%s, amount=%s}", from, to, amount);
        System.out.println("[ASYNC] Published: " + event);
        // Producer does not wait for consumers to finish
    }

    public static void main(String[] args) {
        // Rajesh checks balance: needs immediate answer
        BigDecimal balance = checkBalance("SAV2001");
        System.out.println("Rajesh balance: " + balance);

        System.out.println();

        // Priya transfers money: async is better
        initiateTransfer("SAV3002", "SAV2001", new BigDecimal("25000.00"));
        System.out.println("Priya continues without waiting");
    }
}
TEXT
[SYNC] Fetching balance for SAV2001
Rajesh balance: 75000.00

[ASYNC] Published: TransferInitiated{from=SAV3002, to=SAV2001, amount=25000.00}
Priya continues without waiting

What about message ordering and partitioning?

In Kafka, ordering is guaranteed only within a single partition. If you need all transactions for account SAV2001 to be processed in order, you use the account ID as the message key. Kafka hashes the key to determine the partition, so all messages with the same key go to the same partition.

If you do not set a key, Kafka distributes messages round-robin across partitions, and ordering is lost.

JAVA
package com.bank.messaging;

import java.util.HashMap;
import java.util.Map;
import java.util.ArrayList;
import java.util.List;

public class PartitioningDemo {

    // Simulate 3 partitions
    private static final int PARTITION_COUNT = 3;
    private static final Map<Integer, List<String>> partitions = new HashMap<>();

    static {
        for (int i = 0; i < PARTITION_COUNT; i++) {
            partitions.put(i, new ArrayList<>());
        }
    }

    static void send(String key, String value) {
        int partition = Math.abs(key.hashCode()) % PARTITION_COUNT;
        partitions.get(partition).add(key + ":" + value);
    }

    public static void main(String[] args) {
        // All SAV2001 transactions go to the same partition
        send("SAV2001", "deposit:10000");
        send("SAV2001", "withdraw:5000");
        send("SAV2001", "deposit:20000");

        // SAV3002 transactions may go to a different partition
        send("SAV3002", "deposit:30000");
        send("SAV3002", "withdraw:10000");

        for (Map.Entry<Integer, List<String>> entry : partitions.entrySet()) {
            if (!entry.getValue().isEmpty()) {
                System.out.println("Partition-" + entry.getKey()
                    + ": " + entry.getValue());
            }
        }
    }
}
TEXT
Partition-0: [SAV3002:deposit:30000, SAV3002:withdraw:10000]
Partition-2: [SAV2001:deposit:10000, SAV2001:withdraw:5000, SAV2001:deposit:20000]
FeatureKafkaRabbitMQAmazon SQSApache Pulsar
ModelDistributed commit logTraditional brokerManaged queueDistributed log + queue
OrderingPer-partitionPer-queueBest-effort (FIFO available)Per-partition
RetentionTime or size basedUntil consumed4 days default (max 14)Tiered storage
ReplayYes (reset offset)NoNoYes
ThroughputMillions/secThousands/secThousands/secMillions/sec
LatencyLow (ms)Very low (sub-ms)Medium (10-50 ms)Low (ms)
DeliveryAt-least-once, exactly-onceAt-least-once, at-most-onceAt-least-onceAt-least-once, exactly-once
Best forEvent streaming, audit logsTask queues, RPCServerless, simple queuesMulti-tenant streaming
JAVA
package com.bank.messaging;

public class BrokerComparisonDemo {

    // This code shows conceptual usage patterns, not real clients
    public static void main(String[] args) {
        // Kafka: event streaming for all account transactions
        System.out.println("Kafka: publish to 'account-transactions' topic");
        System.out.println("  - 3 partitions keyed by account ID");
        System.out.println("  - Retained for 7 days for replay");
        System.out.println("  - Consumer groups: processor, fraud, analytics");
        System.out.println();

        // RabbitMQ: task queue for statement generation
        System.out.println("RabbitMQ: publish to 'statement-generation' queue");
        System.out.println("  - 5 workers compete for messages");
        System.out.println("  - Message removed after acknowledgment");
        System.out.println("  - DLQ for failed generations");
        System.out.println();

        // SQS: serverless OTP delivery
        System.out.println("SQS: publish to 'otp-delivery' queue");
        System.out.println("  - Lambda triggers on new messages");
        System.out.println("  - Visibility timeout: 30 seconds");
        System.out.println("  - No infrastructure to manage");
    }
}
TEXT
Kafka: publish to 'account-transactions' topic
  - 3 partitions keyed by account ID
  - Retained for 7 days for replay
  - Consumer groups: processor, fraud, analytics

RabbitMQ: publish to 'statement-generation' queue
  - 5 workers compete for messages
  - Message removed after acknowledgment
  - DLQ for failed generations

SQS: publish to 'otp-delivery' queue
  - Lambda triggers on new messages
  - Visibility timeout: 30 seconds
  - No infrastructure to manage

What about message serialization and schema evolution?

Messages need a format. Common choices are JSON, Avro, and Protobuf. In banking systems, schema evolution matters because producers and consumers are deployed independently.

Avro with a Schema Registry is the standard choice for Kafka in production. It validates schemas at write time and supports backward-compatible changes.

JAVA
package com.bank.messaging;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;
import java.math.BigDecimal;

public class SerializationDemo {

    record TransactionEvent(
        String transactionId,
        String fromAccount,
        String toAccount,
        BigDecimal amount,
        String currency,
        long timestamp
    ) {}

    public static void main(String[] args) throws Exception {
        ObjectMapper mapper = new ObjectMapper();

        // Producer serializes
        TransactionEvent event = new TransactionEvent(
            "TXN-2001", "SAV2001", "SAV3002",
            new BigDecimal("50000.00"), "INR",
            System.currentTimeMillis()
        );
        String json = mapper.writeValueAsString(event);
        System.out.println("Serialized: " + json);

        // Consumer deserializes
        JsonNode node = mapper.readTree(json);
        System.out.println("Transaction ID: " + node.get("transactionId").asText());
        System.out.println("Amount: " + node.get("amount").asText());

        // Schema evolution: new field with default
        // Old consumers ignore unknown fields
        String newJson = json.replace("}",
            ",\"category\":\"NEFT\"}");
        JsonNode newNode = mapper.readTree(newJson);
        System.out.println("Category (new field): "
            + newNode.path("category").asText("UNKNOWN"));
    }
}
TEXT
Serialized: {"transactionId":"TXN-2001","fromAccount":"SAV2001","toAccount":"SAV3002","amount":50000.00,"currency":"INR","timestamp":1719500000000}
Transaction ID: TXN-2001
Amount: 50000.00
Category (new field): NEFT

What should a senior engineer know?

- Back-pressure. When consumers are slower than producers, messages pile up in the broker. Kafka handles this by letting consumers control their own polling rate. RabbitMQ uses prefetch limits to prevent overwhelming a consumer. In a banking system, back-pressure during month-end statement generation can cause millions of messages to queue up.

- Exactly-once semantics. Kafka supports exactly-once processing through idempotent producers (each message has a producer ID and sequence number) and transactional consumers (consume, process, produce, and commit as one atomic operation). For financial transactions, exactly-once is critical.

- Consumer lag monitoring. The difference between the latest offset in a partition and the consumer's committed offset is the consumer lag. If lag grows continuously, consumers cannot keep up. Monitor with Kafka's consumer-groups CLI or tools like Burrow.

- Message ordering across partitions. Kafka guarantees order only within a partition. If a banking workflow requires global ordering (rare), you must use a single partition, which limits throughput to one consumer.

- Compacted topics. Kafka supports log compaction where only the latest value for each key is retained. This is useful for maintaining a current snapshot of account balances without storing the full history.

- Idempotent consumers in production. Use a database table with a unique constraint on the message ID. Before processing, insert the ID. If the insert fails with a duplicate key, skip the message. This is more reliable than an in-memory Set.

What are common interview follow-up questions?

  • What happens if a Kafka consumer crashes before committing its offset?
  • How do you handle message ordering when a topic has multiple partitions?
  • What is the difference between at-most-once, at-least-once, and exactly-once delivery?
  • How do you implement a dead letter queue in Kafka?
  • When would you choose RabbitMQ over Kafka?
  • How do you monitor consumer lag in production?
  • What is the Outbox pattern and why do you need it?

Visual Overview

DIAGRAM
  Queue: Point-to-Point Messaging
  ================================

  Producer        Queue            Consumers
  ────────      ──────────      ────────────
  Rajesh   ──>  │ MSG-1  │ ──>  Consumer-A (processes MSG-1)
  Priya    ──>  │ MSG-2  │ ──>  Consumer-B (processes MSG-2)
  Amit     ──>  │ MSG-3  │ ──>  Consumer-A (processes MSG-3)
                └────────┘
  Each message goes to exactly ONE consumer.


  Topic: Publish-Subscribe Messaging
  ===================================

  Producer        Topic              Subscribers
  ────────      ──────────      ──────────────────
                              ┌─> NotificationService
  Transfer  ──> │ EVENT  │ ──┼─> FraudDetectionService
  Service       └────────┘   └─> StatementService

  Each subscriber gets a COPY of every message.


  Kafka: Consumer Groups
  ========================

  Topic: account-transactions (3 partitions)
  ┌──────────────────────────────────────────────┐
  │  P0: [SAV2001 events]                         │
  │  P1: [SAV3002 events]                         │
  │  P2: [SAV4003 events]                         │
  └──────────────────────────────────────────────┘
           │                    │
           ▼                    ▼
  Group: processor        Group: fraud-detector
  ┌─────────────────┐    ┌──────────────────┐
  │ C1 reads P0, P1 │    │ C3 reads P0      │
  │ C2 reads P2     │    │ C4 reads P1, P2  │
  └─────────────────┘    └──────────────────┘
  Within a group: each partition to one consumer.
  Across groups: every group reads all messages.

Key Takeaway

A message broker is the post office of a distributed system. When Rajesh submits a loan application at the National Bank of India, the clerk does not process it on the spot. The clerk drops it into a tray, and a loan officer picks it up when they are free. That tray is a queue. When the bank manager announces a new interest rate, every department reads the same notice. That notice board is a topic.

Queues and topics solve different problems. A queue ensures each message is processed exactly once by one consumer, which is perfect for tasks like generating monthly statements or sending OTP codes. A topic ensures every subscriber receives every message, which is perfect for events like account balance changes that the notification service, fraud detection, and statement service all need to see.

Kafka combines both patterns with consumer groups. Within a group, each partition goes to one consumer (queue behavior). Across groups, every group reads all messages (topic behavior). The partition key ensures ordering for related messages: all transactions for SAV2001 go to the same partition and are processed in order.

The most critical rule in financial messaging is idempotency. Brokers may redeliver messages after a timeout, a crash, or a network issue. If the consumer blindly processes every delivery, a deposit could be credited twice. Always track processed message IDs and skip duplicates.

One sentence to remember: a queue is one message to one consumer, a topic is one message to all subscribers, and Kafka's consumer groups give you both.

---

Share
Use ← → arrow keys to navigate