Definition
- Multi-datacenter replication -- copies data between independent Kafka clusters in different regions using MirrorMaker 2 or a managed equivalent, providing disaster recovery and geo-locality.
- Active-passive -- one region takes all writes while the other is a warm standby you fail over to during a disaster, which is simpler and avoids conflicts but wastes standby capacity.
- Active-active -- both regions accept writes and replicate to each other, giving lower local latency and no idle capacity, but you must solve offset translation, loop prevention, and conflicting writes.
- MirrorMaker 2 -- built on Kafka Connect, replicates topic data, consumer group offsets, and emits checkpoints so a consumer can resume in the other region near where it left off.
- Topic naming -- MirrorMaker 2 prefixes mirrored topics with the source cluster alias to prevent infinite replication loops in bidirectional setups.
Imagine National Bank of India running two branches in Mumbai and Chennai. Active-passive is like Mumbai doing all the work while Chennai keeps an up-to-date copy of the ledgers in a locked room, ready to take over only if Mumbai floods. Active-active is like both branches serving customers at the same time and constantly couriering copies of their ledgers to each other, which is faster for local customers but risks two branches editing the same account at once and having to reconcile.
Understanding the Concepts
What is multi-datacenter Kafka replication?
Kafka's built-in replication keeps copies within one cluster across brokers, but it does not span regions, because stretching the in-sync replica set across high-latency links would cripple write latency. So cross-region copying uses a separate tool. MirrorMaker 2, built on Kafka Connect, is the standard. It runs source and checkpoint connectors that read from the source cluster and produce into the target cluster.
package com.bank.kafka;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
// Producer in Mumbai region writes to local cluster
public class MumbaiTransferProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "mumbai-kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("transfers", "SAV2001", "DEBIT 50000"));
System.out.println("Rajesh's transfer written to Mumbai cluster");
// MirrorMaker 2 will replicate this to Chennai as 'mumbai.transfers'
}
}
}
Expected output:
Rajesh's transfer written to Mumbai cluster
How does topic naming and loop prevention work?
MirrorMaker 2 by default prefixes mirrored topics with the source cluster alias. A topic transfers from the Mumbai cluster appears as mumbai.transfers in Chennai. This prevents infinite replication loops in active-active, because each cluster only forwards its own locally-originated topics, not the already-prefixed mirrored ones.
# MirrorMaker 2 configuration for Mumbai -> Chennai replication
# mm2.properties
clusters = mumbai, chennai
mumbai.bootstrap.servers = mumbai-kafka-1:9092
chennai.bootstrap.servers = chennai-kafka-1:9092
mumbai->chennai.enabled = true
mumbai->chennai.topics = transfers, settlements, account-events
# In Chennai, topics appear as:
# mumbai.transfers
# mumbai.settlements
# mumbai.account-events
In Chennai cluster:
mumbai.transfers (mirrored from Mumbai)
transfers (local Chennai writes)
chennai.transfers (would appear in Mumbai if bidirectional)
How does offset translation work?
Offsets are per-cluster, so offset 42 in Mumbai is not offset 42 in Chennai. MirrorMaker 2 writes checkpoint records that map source offsets to target offsets for each consumer group. The RemoteClusterUtils API lets a failed-over consumer translate its last committed Mumbai offset into the right Chennai offset.
package com.bank.kafka;
import org.apache.kafka.connect.mirror.RemoteClusterUtils;
import org.apache.kafka.common.TopicPartition;
import java.util.Map;
import java.util.HashMap;
public class OffsetTranslationDemo {
public static void main(String[] args) throws Exception {
Map<String, Object> config = new HashMap<>();
config.put("bootstrap.servers", "chennai-kafka-1:9092");
// Translate Mumbai offsets to Chennai offsets for the settlement group
Map<TopicPartition, Long> translatedOffsets =
RemoteClusterUtils.translateOffsets(
config, "mumbai", "settlement-group", java.time.Duration.ofSeconds(10));
translatedOffsets.forEach((tp, offset) ->
System.out.println(tp + " -> Chennai offset: " + offset));
}
}
Expected output:
mumbai.transfers-0 -> Chennai offset: 45023
mumbai.transfers-1 -> Chennai offset: 44890
What is the active-passive design?
All producers and consumers run against the primary region. MirrorMaker 2 continuously copies topics and offsets to the standby. During a disaster you redirect clients to the standby, translate offsets, and resume.
| Aspect | Active-Passive |
|---|---|
| Writes | Primary region only |
| Standby | Warm, receives continuous replicated data |
| Failover | Deliberate operation with offset translation |
| Data loss window | Equal to replication lag |
| Conflict risk | None -- single writer |
| Capacity utilization | Standby mostly idle |
# Failover procedure for active-passive
# 1. Redirect producers to Chennai
# 2. Translate offsets for consumer groups
kafka-consumer-groups.sh --bootstrap-server chennai-kafka-1:9092 \
--group settlement-service --reset-offsets \
--to-offset 45023 --topic mumbai.transfers --execute
# 3. Start consumers against Chennai cluster
What is the active-active design and what are the hard problems?
Both regions accept writes and mirror to each other. Local users get low latency because they write to their nearest cluster. The hard problems are three: loop prevention, offset translation, and conflicting writes.
For money movement, true multi-master writes to the same account are dangerous, so banks usually pin write ownership of each account to a home region, effectively making it active-passive per key while using both regions fully.
package com.bank.kafka;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
// Active-active with home-region ownership
public class HomeRegionProducer {
private final KafkaProducer<String, String> producer;
private final String homeRegion; // "mumbai" or "chennai"
public HomeRegionProducer(KafkaProducer<String, String> producer, String homeRegion) {
this.producer = producer;
this.homeRegion = homeRegion;
}
public void sendTransfer(String accountId, String payload) {
String accountRegion = getHomeRegion(accountId);
if (!accountRegion.equals(homeRegion)) {
System.out.println("SKIP: Account " + accountId
+ " belongs to " + accountRegion + ", not " + homeRegion);
return; // Route to the correct region instead
}
producer.send(new ProducerRecord<>("transfers", accountId, payload));
System.out.println("Sent transfer for " + accountId + " in " + homeRegion);
}
private String getHomeRegion(String accountId) {
// Rajesh (SAV2001) -> Mumbai, Priya (SAV3001) -> Chennai
return accountId.startsWith("SAV2") ? "mumbai" : "chennai";
}
}
How is active-passive different from active-active?
| Aspect | Active-Passive | Active-Active |
|---|---|---|
| Write regions | One (primary) | Both |
| Latency | Higher for remote users | Low for local users |
| Capacity utilization | Standby idle | Full utilization |
| Conflicts | None | Must handle or prevent |
| Loop prevention | Not needed | Required (topic prefix) |
| Offset translation | On failover only | Continuous |
| Complexity | Simple | High |
| Best for banking | Default choice | With home-region ownership |
# Active-active bidirectional MirrorMaker 2 config
clusters = mumbai, chennai
mumbai->chennai.enabled = true
chennai->mumbai.enabled = true
mumbai->chennai.topics = transfers, settlements
chennai->mumbai.topics = transfers, settlements
# Loop prevention: mumbai.transfers in Chennai is NOT re-mirrored back
What about monitoring and recovery point objectives?
Replication lag defines your recovery point objective (RPO). You monitor MirrorMaker 2 lag closely, and you decide the recovery time objective (RTO) by how fast you can redirect clients and translate offsets.
# Monitor MirrorMaker 2 replication lag
kafka-consumer-groups.sh --bootstrap-server chennai-kafka-1:9092 \
--describe --group mm2-mirror-source-connector
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
mm2-mirror-source-connector transfers 0 45023 45100 77
mm2-mirror-source-connector transfers 1 44890 44950 60
What should a senior engineer know?
For money movement, true multi-master writes to the same account are dangerous, so banks use active-active for read scaling and locality but pin write ownership of each account to a home region. This is effectively active-passive per key while using both regions fully. You must test failover regularly or it will not work when regulators ask. Replication lag should be alerted on, and you should drill the failover quarterly.
Visual Overview
Active-Passive Topology ======================== ┌──────────────────────┐ ┌──────────────────────┐ │ Mumbai Cluster │ │ Chennai Cluster │ │ (Primary) │ │ (Standby) │ │ │ MM2 │ │ │ transfers ─────────────────► mumbai.transfers │ │ settlements ───────────────► mumbai.settlements │ │ account-events ────────────► mumbai.account-events │ │ │ │ │ │ Producers + Consumers│ │ (idle consumers) │ └──────────────────────┘ └──────────────────────┘ Active-Active with Home-Region Ownership ========================================= ┌──────────────────────┐ ┌──────────────────────┐ │ Mumbai Cluster │ MM2 │ Chennai Cluster │ │ │ ◄─────► │ │ │ transfers │ │ transfers │ │ mumbai -> chennai ──────────► │ mumbai.transfers │ │ chennai.transfers ◄────────── │ chennai -> mumbai │ │ │ │ │ │ Rajesh (SAV2001) │ │ Priya (SAV3001) │ │ writes here only │ │ writes here only │ └──────────────────────┘ └──────────────────────┘ Loop Prevention: mumbai.transfers in Chennai is NOT re-mirrored to Mumbai (prefix check prevents infinite loop)
Key Takeaway
Kafka's own replication keeps copies within one cluster but does not span regions, because stretching the in-sync replica set across a high-latency link would cripple write latency. So for multi-datacenter I use a separate copying layer, typically MirrorMaker 2, which replicates topic data, consumer group offsets, and emits checkpoints so a consumer can resume in the other region near where it left off.
The topology choice is active-passive versus active-active. Active-passive is simpler: one region takes all writes and the other is a warm standby that I fail over to during a disaster. Active-active lets both regions accept writes, which gives local users lower latency and uses all capacity, but it introduces three hard problems: loop prevention, offset translation, and conflicting writes to the same key.
For money at National Bank of India, I never allow true multi-master writes to the same account, so I pin each account's write ownership to a home region. Rajesh's account is written only in Mumbai and Priya's only in Chennai, both regions mirror to each other, and the recovery point objective is the mirror lag of a second or two.
Remember one sentence: MirrorMaker 2 copies between clusters with topic prefixes for loop prevention and checkpoint-based offset translation, and for banking you use active-active with home-region ownership to avoid conflicting writes.
---