Definition
- Flag-day cutover – Switching everything at once at midnight; unacceptable for live financial workflows because bugs mean lost or duplicated money.
- Coexistence – Both REST and Kafka paths run simultaneously; the old path remains the source of truth during validation.
- Shadow processing – A new consumer applies events into a parallel copy of state without affecting real customers; a reconciler compares outcomes.
- Feature flags – Control which path is authoritative per segment, enabling gradual ramp from 0% to 100% with instant rollback.
- Reconciliation – Automated comparison of shadow and authoritative outcomes to prove equivalence under production load.
National Bank of India wants to switch from processing transfers at the old manual counter to a new automated conveyor. They run both at once: the old counter still officially processes every transfer, while the new conveyor quietly processes a copy and the staff compare results every evening. Once the conveyor matches the counter perfectly for weeks, they let it handle 10% of real transfers, then 50%, then all, and finally retire the counter. A unique stamp on each slip ensures no transfer is done twice while both run.
Understanding the Concepts
Why is a flag-day cutover unacceptable?
A flag-day cutover switches everything at once. For a live financial workflow, any bug means lost or duplicated money and there is no safe rollback.
package com.bank.migration;
public class FlagDayRisk {
public static void main(String[] args) {
System.out.println("Flag-day cutover risks:");
System.out.println(" - Bug in Kafka consumer -> transfers lost");
System.out.println(" - Duplicate processing -> Rajesh charged twice");
System.out.println(" - No safe rollback: old path already decommissioned");
System.out.println();
System.out.println("Migration principle: reversible stages, always know what is authoritative");
}
}
Expected output:
Flag-day cutover risks:
- Bug in Kafka consumer -> transfers lost
- Duplicate processing -> Rajesh charged twice
- No safe rollback: old path already decommissioned
Migration principle: reversible stages, always know what is authoritative
Stage 1: Coexistence with REST authoritative
The REST endpoint keeps doing its job. Alongside it, publish a Kafka event via a transactional outbox. The event path has no business authority yet.
package com.bank.migration;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class TransferServiceStage1 {
private final TransferRepository transferRepo;
private final OutboxRepository outboxRepo;
public TransferServiceStage1(TransferRepository transferRepo, OutboxRepository outboxRepo) {
this.transferRepo = transferRepo;
this.outboxRepo = outboxRepo;
}
@Transactional
public String processTransfer(String from, String to, BigDecimal amount) {
// REST path remains authoritative
Transfer transfer = transferRepo.save(new Transfer(from, to, amount));
// Outbox: publish event for shadow processing (no business authority)
outboxRepo.save(new OutboxEvent("TRANSFER", transfer.getId(),
"{\"from\":\"" + from + "\",\"to\":\"" + to + "\"}"));
System.out.println("Stage 1: REST authoritative + Kafka event published via outbox");
return transfer.getId();
}
}
Expected output:
Stage 1: REST authoritative + Kafka event published via outbox
Stage 2: Shadow processing with reconciliation
A new idempotent consumer processes events into a shadow table. A reconciliation job compares shadow results with authoritative results.
package com.bank.migration;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class ShadowTransferConsumer {
private final ShadowLedgerRepository shadowRepo;
private final ProcessedMessageRepository processedRepo;
public ShadowTransferConsumer(ShadowLedgerRepository shadowRepo,
ProcessedMessageRepository processedRepo) {
this.shadowRepo = shadowRepo;
this.processedRepo = processedRepo;
}
@KafkaListener(topics = "transfer-events", groupId = "shadow-group")
public void onTransferEvent(String payload) {
String messageId = extractMessageId(payload);
// Idempotent: skip if already processed
if (processedRepo.existsById(messageId)) return;
// Apply to shadow ledger (not real customers)
shadowRepo.applyTransfer(payload);
processedRepo.save(new ProcessedMessage(messageId));
System.out.println("Shadow processed: " + messageId);
}
private String extractMessageId(String payload) { return "MSG-" + payload.hashCode(); }
}
package com.bank.migration;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
public class ReconciliationJob {
@Scheduled(cron = "0 0 2 * * *") // 2 AM daily
public void reconcile() {
// Compare real balances vs shadow balances
// Report any divergence
System.out.println("Reconciliation: comparing real vs shadow balances");
System.out.println("Result: 100% match (0 divergences)");
}
}
Expected output:
Shadow processed: MSG-12345
Reconciliation: comparing real vs shadow balances
Result: 100% match (0 divergences)
Stage 3: Feature flag ramps authority gradually
A feature flag controls which path is authoritative. Start with internal accounts, then 1% of real customers, then ramp.
package com.bank.migration;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class TransferServiceStage3 {
private final FeatureFlagService featureFlags;
public TransferServiceStage3(FeatureFlagService featureFlags) {
this.featureFlags = featureFlags;
}
public String processTransfer(String from, String to, BigDecimal amount) {
if (featureFlags.isKafkaAuthoritative(from)) {
// Kafka path is authoritative for this account
System.out.println("Stage 3: Kafka path is authoritative for " + from);
return publishToKafka(from, to, amount);
} else {
// REST path remains authoritative
System.out.println("Stage 3: REST path is authoritative for " + from);
return processViaRest(from, to, amount);
}
}
private String publishToKafka(String from, String to, BigDecimal amount) {
return "KAFKA-TXN-001";
}
private String processViaRest(String from, String to, BigDecimal amount) {
return "REST-TXN-001";
}
}
Feature flag ramp:
Week 1: internal test accounts only (0.01%)
Week 2: 1% of real customers
Week 3: 10%
Week 4: 50%
Week 5: 100% -> Kafka fully authoritative
At any point: flip flag back -> instant rollback to REST
Stage 4: Decommission the old path
Only after the event path has been fully authoritative and equivalent for a sustained period do you remove the old synchronous business logic.
package com.bank.migration;
public class DecommissionChecklist {
public static void main(String[] args) {
System.out.println("Decommission checklist:");
System.out.println(" [x] Kafka path 100% authoritative for 4+ weeks");
System.out.println(" [x] Reconciliation shows 0 divergences for 4+ weeks");
System.out.println(" [x] Consumer lag < 100ms at peak");
System.out.println(" [x] DLQ volume < 0.001%");
System.out.println(" [x] Shadow comparison code removed");
System.out.println(" [x] Old REST business logic removed");
System.out.println(" [x] Feature flag cleaned up");
}
}
Expected output:
Decommission checklist:
[x] Kafka path 100% authoritative for 4+ weeks
[x] Reconciliation shows 0 divergences for 4+ weeks
[x] Consumer lag < 100ms at peak
[x] DLQ volume < 0.001%
[x] Shadow comparison code removed
[x] Old REST business logic removed
[x] Feature flag cleaned up
Why is idempotency non-negotiable during migration?
During the transition, dual publication or redelivery can cause the same operation to arrive twice. The consumer must deduplicate by message ID.
package com.bank.migration;
public class IdempotencyDuringMigration {
public static void main(String[] args) {
System.out.println("Why idempotency is critical during migration:");
System.out.println(" 1. Outbox relay may publish same event twice (at-least-once)");
System.out.println(" 2. Consumer may crash and get redelivery");
System.out.println(" 3. Both REST and Kafka paths may process briefly");
System.out.println(" -> Without idempotency: Rajesh charged twice");
System.out.println(" -> With idempotency: second attempt is a no-op");
}
}
Expected output:
Why idempotency is critical during migration:
1. Outbox relay may publish same event twice (at-least-once)
2. Consumer may crash and get redelivery
3. Both REST and Kafka paths may process briefly
-> Without idempotency: Rajesh charged twice
-> With idempotency: second attempt is a no-op
What are the safety mechanisms at each stage?
| Stage | Authoritative | Validation | Rollback |
|---|---|---|---|
| 1. Coexistence | REST | None yet | Remove outbox |
| 2. Shadow | REST | Reconciliation vs shadow | Remove shadow consumer |
| 3. Feature flag | Per-account | Reconciliation + monitoring | Flip flag back |
| 4. Decommission | Kafka | Monitoring only | Redeploy old code (emergency) |
Safety guarantees:
- Idempotency: no duplication at any stage
- Outbox: no event loss
- Reconciliation: proves equivalence
- Feature flag: instant rollback control surface
What should a senior engineer know?
The feature flag is the control surface, not the safety mechanism. Idempotency prevents duplication, the outbox prevents loss, and reconciliation gives confidence to advance.
Senior insights:
- Reconciliation must compare at the BALANCE level, not just event counts
- Feature flag should be deterministic (by account_id hash, not random)
- Monitor consumer lag, error rate, DLQ volume at each ramp step
- Keep rollback capability at every stage
- This approach works for any sync-to-async migration, not just REST-to-Kafka
Visual Overview
Migration Stages:
Stage 1: Coexistence (REST authoritative)
┌────────────┐ ┌──────────┐ ┌───────────┐
│ Client │───▶│ REST │───▶│ Database │
│ │ │ (auth) │ │ + Outbox │
└────────────┘ └──────────┘ └─────┬─────┘
│ relay
┌─────▼─────┐
│ Kafka │
│ (no auth) │
└───────────┘
Stage 3: Feature Flag (gradual ramp)
┌────────────┐ ┌──────────────┐
│ Client │───▶│ Feature Flag │
└────────────┘ └──┬───────┬───┘
│ REST │ Kafka
│ (90%) │ (10%)
▼ ▼
┌──────┐ ┌──────┐
│ Old │ │ New │
│ path │ │ path │
└──────┘ └──────┘
Key Takeaway
A flag-day cutover is unacceptable for a live financial workflow because if anything is wrong you have already lost or duplicated money with no safe rollback. Instead, migrate as a sequence of reversible stages where the synchronous REST path and the new Kafka event path coexist and you always know which one is authoritative.
The stages are concrete. First, coexistence: REST stays authoritative while you publish Kafka events via a transactional outbox. Second, shadow processing: an idempotent consumer builds a parallel state and a reconciler compares outcomes daily. Third, feature flags ramp authority from internal accounts to 1% to 100% of real traffic, with instant rollback at every step. Fourth, decommission the old path only after the event path has been fully authoritative and clean for weeks.
At National Bank of India, Pankaj moved the transfer workflow this way. REST stayed authoritative while events flowed via an outbox. A shadow consumer built a shadow ledger, and nightly reconciliation proved equivalence including absorbing duplicate deliveries. The feature flag ramped from internal accounts to 1% over two weeks, with the ability to flip back instantly, and only after 100% clean for a month did they remove the old path.
Remember one sentence: avoid a flag day by running both paths with the old one authoritative, shadow-validate with idempotent consumers and reconciliation, then use feature flags to ramp authority gradually with instant rollback, so ownership transfers without ever losing or duplicating a business operation.
---