How Kafka guarantees ordering within a partition, and multiple consumers | Kafka & Messaging

Q: How does producer idempotence preserve ordering during retries?

With retries enabled and more than one inflight request per connection, a failedthenretried batch could theoretically be reordered. To prevent this, you enable idempotence with enable.idempotence=true, which Kafka sets by default in modern versions. The broker uses a producer id and sequence numbers

Definition

Order is guaranteed per partition because records get sequential offsets and exactly one consumer per group reads each partition.

Partition ordering – Kafka guarantees that records within a single partition are read in the exact order they were written, because each record receives a sequential offset and the log is immutable.
One-consumer-per-partition rule – within a consumer group, Kafka assigns each partition to exactly one consumer, ensuring no two consumers in the same group read the same partition concurrently.
Message key – the value Kafka hashes to choose the target partition, so all records with the same key land in the same partition and stay ordered relative to each other.
Producer idempotence – the enable.idempotence=true setting that makes the broker detect and discard duplicate sends from retries, preserving order even with multiple in-flight requests.
Cross-partition ordering – Kafka makes no ordering guarantee across different partitions; global ordering requires a single-partition topic at the cost of parallelism.

Imagine one drawer of the National Bank of India ledger where pages are glued in sequence: page 0, page 1, page 2. Nobody can shuffle them. Only one clerk is ever allowed to read that drawer for a given team, so the clerk reads page 0, then page 1, then page 2, exactly in order. If the bank added a second clerk to the same team, Kafka would not let both read the same drawer; it hands each clerk a different drawer so there is never a race over the same pages.

Understanding the Concepts

How does Kafka guarantee ordering within a partition?

Ordering starts at the producer. When a producer sends records to a partition, Kafka appends them in send order and assigns each one the next sequential offset. The partition is immutable: once a record sits at offset 5, nothing can insert a record before it. A consumer always reads a partition sequentially from a starting offset forward, so it observes records in the same order the producer wrote them.

JAVA

package com.bank.ordering;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class OrderedTransactionConsumer {

    @KafkaListener(topics = "transactions", groupId = "settlement-group")
    public void processTransaction(ConsumerRecord<String, String> record) {
        System.out.println("Partition " + record.partition()
            + " offset " + record.offset()
            + " key=" + record.key()
            + " value=" + record.value());
        // Within partition 2, offsets are always 0, 1, 2, 3... in order
    }
}

TEXT

Expected output:
Partition 2 offset 0 key=SAV2001 value={"type":"DEBIT","amount":10000}
Partition 2 offset 1 key=SAV2001 value={"type":"CHECK","amount":0}
Partition 2 offset 2 key=SAV2001 value={"type":"CREDIT","amount":5000}
-> Strictly ordered within partition 2

How does the message key control ordering?

The partition a record lands in is decided by its key. If you produce with a key, Kafka hashes the key to choose a partition, so all records with the same key go to the same partition and are therefore ordered relative to one another. In banking, keying by account id guarantees Rajesh's debit, balance check, and next debit are read in the exact order they occurred.

JAVA

package com.bank.producer;

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;

@Service
public class KeyedTransactionProducer {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public KeyedTransactionProducer(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishTransaction(String accountId, String type,
                                   BigDecimal amount) {
        String event = "{\"accountId\":\"" + accountId
            + "\",\"type\":\"" + type
            + "\",\"amount\":" + amount + "}";
        kafkaTemplate.send("transactions", accountId, event);
        System.out.println("Sent " + type + " for " + accountId);
    }
}

TEXT

Expected output:
Sent DEBIT for SAV2001   -> partition 2, offset 0
Sent CHECK for SAV2001   -> partition 2, offset 1  (same key = same partition)
Sent CREDIT for SAV2001  -> partition 2, offset 2  (ordered)
Sent DEBIT for SAV3002   -> partition 5, offset 0  (different key = different partition)

How does the one-consumer-per-partition rule work?

Within a consumer group, Kafka assigns each partition to exactly one consumer. It never splits a partition between two consumers of the same group. So if a topic has three partitions and the group has three consumers, each consumer owns one partition and processes it in order. This is precisely what preserves ordering under parallelism.

JAVA

package com.bank.consumer;

public class ConsumerPartitionAssignment {

    public static void main(String[] args) {
        System.out.println("Topic: transactions (6 partitions)");
        System.out.println("Group: settlement-group");
        System.out.println();
        System.out.println("3 consumers in group:");
        System.out.println("  Consumer A -> P0, P1 (processes both in order)");
        System.out.println("  Consumer B -> P2, P3");
        System.out.println("  Consumer C -> P4, P5");
        System.out.println();
        System.out.println("6 consumers in group:");
        System.out.println("  Consumer A -> P0  |  Consumer D -> P3");
        System.out.println("  Consumer B -> P1  |  Consumer E -> P4");
        System.out.println("  Consumer C -> P2  |  Consumer F -> P5");
        System.out.println();
        System.out.println("8 consumers in group (only 6 partitions):");
        System.out.println("  Consumers A-F: active (one partition each)");
        System.out.println("  Consumer G: IDLE | Consumer H: IDLE");
    }
}

TEXT

Expected output:
Topic: transactions (6 partitions)
Group: settlement-group

3 consumers in group:
  Consumer A -> P0, P1 (processes both in order)
  Consumer B -> P2, P3
  Consumer C -> P4, P5

6 consumers in group:
  Consumer A -> P0  |  Consumer D -> P3
  Consumer B -> P1  |  Consumer E -> P4
  Consumer C -> P2  |  Consumer F -> P5

8 consumers in group (only 6 partitions):
  Consumers A-F: active (one partition each)
  Consumer G: IDLE | Consumer H: IDLE

How does producer idempotence preserve ordering during retries?

With retries enabled and more than one in-flight request per connection, a failed-then-retried batch could theoretically be reordered. To prevent this, you enable idempotence with enable.idempotence=true, which Kafka sets by default in modern versions. The broker uses a producer id and sequence numbers to detect duplicates and preserve order even across retries.

JAVA

package com.bank.config;

import java.util.Properties;

public class IdempotentProducerConfig {

    public static Properties createConfig() {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");

        // Idempotence ensures ordering even with retries
        props.put("enable.idempotence", "true");
        props.put("acks", "all");
        props.put("retries", "3");
        // max.in.flight.requests.per.connection <= 5 with idempotence
        props.put("max.in.flight.requests.per.connection", "5");

        System.out.println("Idempotent producer configured");
        System.out.println("Ordering preserved even during retries");
        return props;
    }

    public static void main(String[] args) {
        createConfig();
    }
}

TEXT

Expected output:
Idempotent producer configured
Ordering preserved even during retries

Without idempotence:
  Batch 1 sent -> fails -> retried
  Batch 2 sent -> succeeds
  Batch 1 retry -> succeeds
  Result: Batch 2 before Batch 1 (REORDERED)

With idempotence:
  Batch 1 sent -> fails -> retried
  Batch 2 sent -> broker holds until Batch 1 completes
  Result: Batch 1 before Batch 2 (ORDERED)

How does ordering work across consumer groups?

Across different consumer groups, each group gets its own independent assignment of all partitions. The fraud group and the audit group both read every partition, each in order, without affecting each other. One group falling behind or replaying from an old offset has zero effect on another group.

JAVA

package com.bank.consumer;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class MultiGroupConsumers {

    @KafkaListener(topics = "transactions", groupId = "fraud-group")
    public void fraudConsumer(String event) {
        System.out.println("[fraud-group] Processing: " + event);
    }

    @KafkaListener(topics = "transactions", groupId = "audit-group")
    public void auditConsumer(String event) {
        System.out.println("[audit-group] Processing: " + event);
    }

    // Both groups read ALL partitions independently
    // Each group has its own offsets
    // fraud-group can be on offset 500 while audit-group is on offset 300
}

TEXT

Expected output:
[fraud-group] Processing: {"accountId":"SAV2001",...}
[audit-group] Processing: {"accountId":"SAV2001",...}
-> Same event processed by both groups independently
-> Each group tracks its own offsets in __consumer_offsets

What are common mistakes with ordering?

The most common mistake is assuming global ordering across partitions. Kafka only guarantees ordering within a partition. If you need total ordering, you must use a single partition, which caps throughput to one consumer.

JAVA

package com.bank.mistakes;

// WRONG: Assuming events from different accounts are ordered
// across partitions. They are NOT.
public class WrongOrderingAssumption {

    public static void main(String[] args) {
        System.out.println("WRONG assumption:");
        System.out.println("  SAV2001 DEBIT at 10:00 (partition 2, offset 0)");
        System.out.println("  SAV3002 CREDIT at 10:01 (partition 5, offset 0)");
        System.out.println("  Consumer may see SAV3002 CREDIT before SAV2001 DEBIT");
        System.out.println("  Cross-partition ordering is NOT guaranteed!");
        System.out.println();
        System.out.println("CORRECT understanding:");
        System.out.println("  SAV2001 DEBIT  (partition 2, offset 0)");
        System.out.println("  SAV2001 CREDIT (partition 2, offset 1)");
        System.out.println("  Consumer ALWAYS sees DEBIT before CREDIT");
        System.out.println("  Within partition 2, ordering is guaranteed.");
    }
}

TEXT

Expected output:
WRONG assumption:
  SAV2001 DEBIT at 10:00 (partition 2, offset 0)
  SAV3002 CREDIT at 10:01 (partition 5, offset 0)
  Consumer may see SAV3002 CREDIT before SAV2001 DEBIT
  Cross-partition ordering is NOT guaranteed!

CORRECT understanding:
  SAV2001 DEBIT  (partition 2, offset 0)
  SAV2001 CREDIT (partition 2, offset 1)
  Consumer ALWAYS sees DEBIT before CREDIT
  Within partition 2, ordering is guaranteed.

What are the edge cases with ordering?

One edge case is increasing partitions later. When you add partitions, the key-to-partition hash mapping changes, so a key that was on partition 2 may now land on partition 4. This breaks per-key ordering for existing keys because new records go to a different partition than the old ones.

Another edge case is producing without a key, which means records are distributed across partitions in a balanced way but with no ordering guarantee for any entity.

JAVA

package com.bank.edge;

public class OrderingEdgeCases {

    public static void main(String[] args) {
        System.out.println("Edge Case 1: Repartitioning");
        System.out.println("  Before: 6 partitions, SAV2001 -> partition 2");
        System.out.println("  After:  12 partitions, SAV2001 -> partition 8");
        System.out.println("  Old records in P2, new records in P8");
        System.out.println("  Ordering broken for SAV2001!");
        System.out.println();
        System.out.println("Edge Case 2: No key");
        System.out.println("  kafkaTemplate.send('transactions', event)");
        System.out.println("  Record goes to random partition");
        System.out.println("  No ordering guarantee for any account");
        System.out.println();
        System.out.println("Edge Case 3: Single partition for global ordering");
        System.out.println("  1 partition = total ordering but max 1 consumer");
        System.out.println("  Use only when global ordering is mandatory");
    }
}

TEXT

Expected output:
Edge Case 1: Repartitioning
  Before: 6 partitions, SAV2001 -> partition 2
  After:  12 partitions, SAV2001 -> partition 8
  Old records in P2, new records in P8
  Ordering broken for SAV2001!

Edge Case 2: No key
  Record goes to random partition, no ordering guarantee

Edge Case 3: Single partition for global ordering
  1 partition = total ordering but max 1 consumer

What should a senior engineer know about ordering?

You trade global ordering for throughput, and the key is your lever. True total ordering requires a single partition, which caps throughput to one consumer. Most banking use cases only need per-entity ordering, so keying by account id gives both order where it matters and parallelism everywhere else. The combination of key-based partitioning and the one-consumer-per-partition rule is what makes Kafka both parallel and ordered.

JAVA

package com.bank.senior;

public class OrderingSeniorSummary {

    public static void main(String[] args) {
        System.out.println("Senior Ordering Checklist:");
        System.out.println("1. Key by entity (accountId) for per-entity ordering");
        System.out.println("2. enable.idempotence=true to prevent retry reordering");
        System.out.println("3. Plan partition count upfront (cannot reduce later)");
        System.out.println("4. One partition per group consumer (never shared)");
        System.out.println("5. Global ordering = 1 partition = 1 consumer (last resort)");
        System.out.println("6. Monitor consumer lag per partition for ordering health");
    }
}

TEXT

Expected output:
Senior Ordering Checklist:
1. Key by entity (accountId) for per-entity ordering
2. enable.idempotence=true to prevent retry reordering
3. Plan partition count upfront (cannot reduce later)
4. One partition per group consumer (never shared)
5. Global ordering = 1 partition = 1 consumer (last resort)
6. Monitor consumer lag per partition for ordering health

Visual Overview

DIAGRAM

Ordering Guarantee Flow
========================

  Producer (key = SAV2001)
      │
      │  send(key="SAV2001", DEBIT)
      │  send(key="SAV2001", CHECK)
      │  send(key="SAV2001", CREDIT)
      │
      ▼
  hash("SAV2001") % 6 = 2
      │
      ▼
  Partition 2:
  ┌──────┬──────┬──────┐
  │ off=0│ off=1│ off=2│
  │DEBIT │CHECK │CREDIT│
  └──────┴──────┴──────┘
      │
      ▼
  Consumer C (settlement-group)
  Reads: DEBIT -> CHECK -> CREDIT (strict order)

  Within Group (one consumer per partition):
  ┌────────────┐  ┌────────────┐  ┌────────────┐
  │ Consumer A │  │ Consumer B │  │ Consumer C │
  │  reads P0  │  │  reads P1  │  │  reads P2  │
  │  reads P3  │  │  reads P4  │  │  reads P5  │
  └────────────┘  └────────────┘  └────────────┘
  No two consumers share a partition = ordering preserved

  Across Groups:
  fraud-group:  reads ALL partitions (own offsets)
  audit-group:  reads ALL partitions (own offsets)
  -> Independent, no interference

Key Takeaway

Kafka guarantees message ordering within a partition because each partition is a strictly ordered, append-only, immutable log. When a producer sends records to a partition, Kafka appends them in send order and assigns each a sequential offset. A consumer reads the partition sequentially, so it sees records in exactly the order they were written. This ordering is guaranteed only within a single partition, never across partitions.

The partition a record lands in is chosen by hashing its key. By keying events by account id at National Bank of India, all of Rajesh's transactions land in the same partition and are read in the exact order they happened: his debit always comes before his balance check. The one-consumer-per-partition rule preserves this under parallelism: within a consumer group, each partition is assigned to exactly one consumer, so no race condition can break the ordering.

Producer idempotence prevents retries from reordering records. Across different consumer groups, each group gets its own independent assignment and reads all partitions without interfering with other groups. The trade-off is that global ordering requires a single partition, which caps throughput to one consumer.

Remember one sentence: Kafka guarantees ordering within a partition through sequential offsets and the one-consumer-per-partition rule, and the message key is your lever for controlling which records stay ordered together.

---