Definition
- Same group id – configuring all service instances with an identical
group.idso Kafka treats them as one consumer group and divides partitions among them. - Competing consumers – the pattern where multiple consumers share the same group id and compete for messages, with each message going to only one consumer.
- Broadcast pattern – the opposite pattern where each consumer has a unique group id, so every consumer receives every message.
- Partition count – the maximum number of active consumers in a group; instances beyond this number sit idle as hot standbys.
- Idempotent consumer – a consumer that produces the same result whether it processes a message once or multiple times, protecting against rare redelivery after rebalances.
Imagine three clerks at National Bank of India all employed on the same team to process incoming transfer slips. Because they are one team sharing one job, the supervisor hands each slip to only one clerk; the other two never see it. That is a single consumer group. If, by mistake, each clerk were treated as a separate one-person team, the supervisor would photocopy every slip and give a copy to all three, and the same transfer would be processed three times.
Understanding the Concepts
How do you ensure only one instance consumes a message?
The behavior hinges entirely on the group id. In Spring Boot you set it once with spring.kafka.consumer.group-id=transfer-processor, and you deploy three instances of the service, all reading this same property. Because they share the group id, Kafka's group coordinator treats them as one group and divides the topic's partitions among them. A record sits in exactly one partition, and that partition is owned by exactly one of the three instances.
package com.bank.consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class TransferProcessor {
// All 3 instances share this group id
@KafkaListener(topics = "transfers", groupId = "transfer-processor")
public void onTransfer(ConsumerRecord<String, String> record) {
System.out.println("Instance " + getInstanceId()
+ " processing transfer: key=" + record.key()
+ " partition=" + record.partition());
}
private String getInstanceId() {
return System.getenv().getOrDefault("INSTANCE_ID", "local");
}
}
Expected output (3 instances, 3 partitions):
Instance A processing transfer: key=SAV2001 partition=2
-> Only Instance A sees this message (partition 2 assigned to A)
-> Instance B and C never see it
How does partition assignment work with three instances?
Suppose the topic transfers has three partitions and you run three instances with the same group id. Kafka assigns partition 0 to instance A, partition 1 to instance B, and partition 2 to instance C. When the producing service sends a transfer keyed by account id SAV2001, Kafka hashes the key to partition 2, so only instance C consumes it.
package com.bank.assignment;
public class ThreeInstanceAssignment {
public static void main(String[] args) {
System.out.println("Topic: transfers (3 partitions)");
System.out.println("Group: transfer-processor (3 instances)");
System.out.println();
System.out.println("Assignment:");
System.out.println(" Instance A -> Partition 0");
System.out.println(" Instance B -> Partition 1");
System.out.println(" Instance C -> Partition 2");
System.out.println();
System.out.println("Producer sends: key=SAV2001");
System.out.println(" hash(SAV2001) % 3 = 2");
System.out.println(" -> Partition 2 -> Instance C processes it");
System.out.println(" -> Instances A and B never see this message");
}
}
Expected output:
Instance C processes transfer for SAV2001
Instances A and B never see it
What is the common mistake with group ids?
The most dangerous mistake is letting each instance generate a unique group id, for example by appending the hostname or a random suffix. That turns each instance into its own group, and since every group receives all partitions, all three instances consume the same message.
package com.bank.mistakes;
import java.util.UUID;
// WRONG: Each instance generates a unique group id.
// All three instances process every message.
public class WrongGroupId {
public static void main(String[] args) {
String instanceA = "transfer-processor-" + UUID.randomUUID();
String instanceB = "transfer-processor-" + UUID.randomUUID();
String instanceC = "transfer-processor-" + UUID.randomUUID();
System.out.println("WRONG group ids:");
System.out.println(" Instance A: " + instanceA);
System.out.println(" Instance B: " + instanceB);
System.out.println(" Instance C: " + instanceC);
System.out.println();
System.out.println("Each instance is its OWN group!");
System.out.println("All three process EVERY message.");
System.out.println("Result: Rajesh debited THREE times!");
}
}
package com.bank.mistakes;
// CORRECT: All instances use the same group id.
public class CorrectGroupId {
public static void main(String[] args) {
String groupId = "transfer-processor";
System.out.println("CORRECT group id: " + groupId);
System.out.println(" Instance A: " + groupId);
System.out.println(" Instance B: " + groupId);
System.out.println(" Instance C: " + groupId);
System.out.println();
System.out.println("All three share ONE group.");
System.out.println("Each message processed by exactly ONE instance.");
System.out.println("Result: Rajesh debited exactly ONCE.");
}
}
WRONG: unique group ids = triple processing = triple debit
CORRECT: shared group id = single processing = one debit
What happens if there are fewer partitions than instances?
If the topic has only one partition but you run three instances in the same group, only one instance will ever be active because a partition can be owned by just one consumer. The other two sit idle as hot standbys. They still provide failover: if the active instance dies, a rebalance hands the partition to one of the idle instances.
package com.bank.scaling;
public class PartitionCountVsInstances {
public static void main(String[] args) {
System.out.println("1 partition, 3 instances:");
System.out.println(" Instance A -> Partition 0 (ACTIVE)");
System.out.println(" Instance B -> (IDLE)");
System.out.println(" Instance C -> (IDLE)");
System.out.println(" Only Instance A processes messages");
System.out.println();
System.out.println("If Instance A dies:");
System.out.println(" Rebalance -> Instance B -> Partition 0 (ACTIVE)");
System.out.println(" Failover works, but no parallel processing");
System.out.println();
System.out.println("3 partitions, 3 instances:");
System.out.println(" All three ACTIVE, processing in parallel");
System.out.println(" Full utilization of all instances");
}
}
Expected output:
1 partition, 3 instances: 1 active, 2 idle
3 partitions, 3 instances: 3 active, 0 idle
How do you handle competing consumers vs broadcast?
For competing consumers where each message is processed once, keep the group id identical across instances. For broadcast where every instance must see every message, give each instance a distinct group id.
package com.bank.patterns;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class CompetingVsBroadcast {
// Competing consumers: each transfer processed by ONE instance
@KafkaListener(topics = "transfers", groupId = "transfer-processor")
public void competingConsumer(String event) {
System.out.println("[competing] Processed once: " + event);
}
// Broadcast: each instance gets every config update
// group id includes instance id for uniqueness
@KafkaListener(topics = "config-updates",
groupId = "#{T(java.lang.System).getenv('HOSTNAME')}-config")
public void broadcastConsumer(String event) {
System.out.println("[broadcast] All instances see: " + event);
}
}
Expected output:
Competing (transfers): one instance processes each transfer
Broadcast (config-updates): all instances receive every config update
Why do you need consumer idempotency even with same group id?
Even within a single group, a rebalance after a crash can cause a message to be redelivered to a different instance if the offset was not committed before the crash. This gives at-least-once delivery. The consumer should be idempotent so that a rare redelivery does not double-debit the account.
package com.bank.service;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class IdempotentTransferProcessor {
private final TransferRepository transferRepo;
public IdempotentTransferProcessor(TransferRepository transferRepo) {
this.transferRepo = transferRepo;
}
public void processTransfer(String transferId, String accountId,
BigDecimal amount) {
// Idempotency check: skip if already processed
if (transferRepo.existsByTransferId(transferId)) {
System.out.println("Skipping duplicate transfer: " + transferId);
return;
}
// Process the transfer
transferRepo.save(transferId, accountId, amount);
System.out.println("Processed transfer: " + transferId
+ " amount=" + amount);
}
}
interface TransferRepository {
boolean existsByTransferId(String transferId);
void save(String transferId, String accountId, BigDecimal amount);
}
Expected output (first delivery):
Processed transfer: TRF-001 amount=50000.00
Expected output (redelivery after rebalance):
Skipping duplicate transfer: TRF-001
-> Rajesh never double-debited
What about Spring Boot configuration?
In Spring Boot, the group id is a single property in application.yml. Every instance deployed with this configuration shares the group id automatically.
package com.bank.config;
public class SpringBootGroupIdConfig {
public static void main(String[] args) {
System.out.println("application.yml:");
System.out.println("spring:");
System.out.println(" kafka:");
System.out.println(" consumer:");
System.out.println(" group-id: transfer-processor");
System.out.println(" auto-offset-reset: earliest");
System.out.println(" enable-auto-commit: false");
System.out.println();
System.out.println("Deploy 3 instances with same config:");
System.out.println(" java -jar bank-service.jar --server.port=8081");
System.out.println(" java -jar bank-service.jar --server.port=8082");
System.out.println(" java -jar bank-service.jar --server.port=8083");
System.out.println();
System.out.println("All 3 share group-id=transfer-processor");
System.out.println("Kafka divides partitions automatically");
}
}
Expected output:
application.yml:
spring.kafka.consumer.group-id: transfer-processor
Deploy 3 instances -> all share group-id -> partitions divided
What should a senior engineer know?
The senior insight is that same-group-id guarantees single consumption in the normal case, and idempotency guarantees single effect even across the rare redelivery during failover. The combination of shared group id plus idempotent consumer is the standard pattern for financial message processing.
package com.bank.senior;
public class SingleConsumptionSummary {
public static void main(String[] args) {
System.out.println("Ensuring single consumption:");
System.out.println("1. Same group.id across all instances");
System.out.println(" -> Kafka assigns each partition to one instance");
System.out.println("2. Idempotent consumer (unique constraint + check)");
System.out.println(" -> Protects against rare redelivery");
System.out.println("3. Sufficient partitions (>= instance count)");
System.out.println(" -> All instances active (not idle)");
System.out.println("4. Manual offset commit (enable.auto.commit=false)");
System.out.println(" -> At-least-once delivery");
System.out.println();
System.out.println("Normal case: group id handles it");
System.out.println("Edge case: idempotency handles redelivery");
}
}
Expected output:
Ensuring single consumption:
1. Same group.id -> one instance per partition
2. Idempotent consumer -> no duplicate effects
3. Sufficient partitions -> all instances active
4. Manual offset commit -> at-least-once delivery
Visual Overview
Same Group ID = Single Consumption
===================================
Producer Service
│
│ send("transfers", key="SAV2001", event)
│
▼
Kafka Topic: transfers (3 partitions)
┌──────────┬──────────┬──────────┐
│ P0 │ P1 │ P2 │
└────┬─────┘────┬─────┘────┬─────┘
│ │ │
▼ ▼ ▼
Group: transfer-processor
┌──────────┐ ┌──────────┐ ┌──────────┐
│Instance A│ │Instance B│ │Instance C│
│ -> P0 │ │ -> P1 │ │ -> P2 │
└──────────┘ └──────────┘ └──────────┘
key=SAV2001 -> hash -> P2 -> Instance C ONLY
WRONG (unique group ids):
┌──────────┐ ┌──────────┐ ┌──────────┐
│Group A │ │Group B │ │Group C │
│ALL P0-P2 │ │ALL P0-P2 │ │ALL P0-P2 │
└──────────┘ └──────────┘ └──────────┘
All 3 instances process EVERY message = triple debit!
Key Takeaway
To ensure only one of three service instances consumes a given Kafka message, configure all three instances with the same consumer group id. Within a single consumer group, Kafka assigns each partition to exactly one consumer, so a record that lives in one partition is delivered to only one instance. In Spring Boot this is a single property: spring.kafka.consumer.group-id=transfer-processor, set identically on every instance.
The classic mistake is letting each instance generate a unique group id, which turns each instance into its own group and causes every message to be processed by all three instances. At National Bank of India, this would mean Rajesh gets debited three times for a single transfer. The fix is always the same: shared group id for competing consumers, unique group id only for broadcast patterns like configuration updates.
To use all three instances in parallel, the topic needs at least three partitions. With fewer partitions, some instances sit idle as hot standbys, which still provides failover. Even with the correct group id, a rebalance after a crash can cause rare redelivery, so the consumer must be idempotent: a unique constraint on the transfer id and a processed-id check ensure Rajesh is never double-debited.
Remember one sentence: same group id guarantees single consumption in the normal case, and idempotency guarantees single effect even during rare redelivery after a failover.
---