Definition
- Code rollback – Redeploying the previous binary; trivially easy in CI/CD but only restores the old system if state is compatible.
- Schema-incompatible data – The new version writes a new column or format; the old code cannot read it after rollback.
- Unrecallable events – Events published to Kafka before rollback are permanent; the old consumer may not parse them.
- Irreversible side effects – Money wired, emails sent, or partner APIs called cannot be un-done by any code change.
- Expand-and-contract – Keep old and new schemas compatible throughout the change: add before remove, dual-write, dual-publish.
At National Bank of India, rolling back is like telling yesterday's clerk to take over after today's new clerk caused problems. That works only if nothing permanent changed. But suppose today's clerk started writing entries in a new ledger format, sent payment-done letters to partner banks, and wired real money. Bringing back yesterday's clerk does not undo any of that: the old clerk cannot read the new ledger, the partners have acted on the letters, and the money is gone.
Understanding the Concepts
Why does code rollback seem easy but fail in distributed systems?
In a monolith with one database, rollback feels simple: redeploy the old binary. But even there, the hidden assumption is that the old code can still read the current state.
package com.bank.rollback;
public class RollbackIllusion {
public static void main(String[] args) {
System.out.println("The illusion: 'Just redeploy the old version'");
System.out.println();
System.out.println("Reality: rollback = code rollback + state rollback");
System.out.println(" Code rollback: trivial (deploy previous artifact)");
System.out.println(" State rollback: often IMPOSSIBLE");
System.out.println();
System.out.println("Three ways state breaks rollback:");
System.out.println(" 1. Schema-incompatible data");
System.out.println(" 2. Unrecallable events");
System.out.println(" 3. Irreversible external side effects");
}
}
Expected output:
The illusion: 'Just redeploy the old version'
Reality: rollback = code rollback + state rollback
Code rollback: trivial (deploy previous artifact)
State rollback: often IMPOSSIBLE
Three ways state breaks rollback:
1. Schema-incompatible data
2. Unrecallable events
3. Irreversible external side effects
How does schema-incompatible data break rollback?
The new version writes data in a new format. After rollback, the old code reads data it does not understand.
package com.bank.rollback;
import java.math.BigDecimal;
public class SchemaIncompatibility {
// New version writes balance as integer paise in a new column
// Old version expects decimal rupees in the original column
// New code writes: balance_minor = 95000000 (paise)
// balance = NULL (stopped writing it)
// After rollback, old code reads:
// BigDecimal balance = row.getBigDecimal("balance"); // NULL!
// -> Shows Rs. 0 to Rajesh, or NullPointerException
public static void main(String[] args) {
System.out.println("Schema incompatibility:");
System.out.println(" New code: writes balance_minor (paise), stops writing balance (rupees)");
System.out.println(" Rollback: old code reads balance -> NULL");
System.out.println(" Result: Rajesh sees Rs. 0 or app crashes");
System.out.println();
System.out.println(" The CODE went back. The DATA did not.");
}
}
Expected output:
Schema incompatibility:
New code: writes balance_minor (paise), stops writing balance (rupees)
Rollback: old code reads balance -> NULL
Result: Rajesh sees Rs. 0 or app crashes
The CODE went back. The DATA did not.
How do unrecallable events break rollback?
Events published to Kafka before rollback are permanent. Rolling the producer back does not recall them.
package com.bank.rollback;
public class UnrecallableEvents {
public static void main(String[] args) {
System.out.println("Unrecallable events:");
System.out.println(" New producer publishes TransferCompletedV2 events to Kafka");
System.out.println(" 10,000 V2 events committed before bug discovered");
System.out.println(" Roll producer back to V1");
System.out.println();
System.out.println(" Problem: 10,000 V2 events STILL in Kafka topic");
System.out.println(" Old V1 consumers cannot parse V2 format");
System.out.println(" In event-sourced systems: V2 events are permanent history");
System.out.println(" No binary rollback can un-publish committed messages");
}
}
Expected output:
Unrecallable events:
New producer publishes TransferCompletedV2 events to Kafka
10,000 V2 events committed before bug discovered
Roll producer back to V1
Problem: 10,000 V2 events STILL in Kafka topic
Old V1 consumers cannot parse V2 format
In event-sourced systems: V2 events are permanent history
No binary rollback can un-publish committed messages
How do irreversible side effects break rollback?
Money wired, emails sent, or partner APIs called cannot be un-done by deploying old code.
package com.bank.rollback;
public class IrreversibleSideEffects {
public static void main(String[] args) {
System.out.println("Irreversible actions the new version performed:");
System.out.println(" 1. Wired Rs. 50,000 to Priya's bank via NEFT");
System.out.println(" 2. Sent 'Transfer complete' SMS to Rajesh");
System.out.println(" 3. Called partner bank's settlement API");
System.out.println();
System.out.println("After code rollback:");
System.out.println(" Money is GONE from the vault");
System.out.println(" SMS is SENT");
System.out.println(" Partner bank has ACTED on the settlement");
System.out.println();
System.out.println("No binary rollback can un-send money or un-send a notification");
}
}
Expected output:
Irreversible actions the new version performed:
1. Wired Rs. 50,000 to Priya's bank via NEFT
2. Sent 'Transfer complete' SMS to Rajesh
3. Called partner bank's settlement API
After code rollback:
Money is GONE from the vault
SMS is SENT
Partner bank has ACTED on the settlement
No binary rollback can un-send money or un-send a notification
Why must rollback be designed in, not discovered during a fire?
The window in which rollback is safe closes the instant the first incompatible write, unrecallable event, or irreversible side effect occurs.
package com.bank.rollback;
public class DesignTimeProperty {
public static void main(String[] args) {
System.out.println("Rollback safety timeline:");
System.out.println(" t=0: deploy new version");
System.out.println(" t=1s: first incompatible data written -> rollback window CLOSING");
System.out.println(" t=5s: first V2 event published -> rollback window SMALLER");
System.out.println(" t=10s: first money wired -> rollback window CLOSED");
System.out.println();
System.out.println("By the time you discover the bug (t=30min):");
System.out.println(" All three dimensions are already irreversible");
System.out.println(" 'Just roll back' no longer works");
System.out.println();
System.out.println("Solution: plan the backward path BEFORE shipping");
}
}
Expected output:
Rollback safety timeline:
t=0: deploy new version
t=1s: first incompatible data written -> rollback window CLOSING
t=5s: first V2 event published -> rollback window SMALLER
t=10s: first money wired -> rollback window CLOSED
By the time you discover the bug (t=30min):
All three dimensions are already irreversible
'Just roll back' no longer works
Solution: plan the backward path BEFORE shipping
How does expand-and-contract make changes reversible?
Keep old and new states compatible throughout the change window. Add before remove. Dual-write so the old code can still read.
package com.bank.rollback;
public class ExpandAndContract {
public static void main(String[] args) {
System.out.println("Expand-and-contract for DATA:");
System.out.println(" Phase 1 (expand): add balance_minor column, dual-write BOTH columns");
System.out.println(" Phase 2 (migrate): consumers switch to balance_minor");
System.out.println(" Phase 3 (contract): stop writing balance, drop column");
System.out.println(" Rollback at Phase 1: old code reads balance column -> still there!");
System.out.println();
System.out.println("Expand-and-contract for EVENTS:");
System.out.println(" Phase 1: dual-publish V1 and V2 events");
System.out.println(" Phase 2: consumers migrate to V2");
System.out.println(" Phase 3: stop publishing V1");
System.out.println(" Rollback at Phase 1: old consumers read V1 -> still published!");
System.out.println();
System.out.println("Expand-and-contract for SIDE EFFECTS:");
System.out.println(" Hold irreversible action in outbox behind feature flag");
System.out.println(" Disable flag -> action stops instantly, no redeploy needed");
}
}
Expected output:
Expand-and-contract for DATA:
Phase 1 (expand): add balance_minor column, dual-write BOTH columns
Phase 2 (migrate): consumers switch to balance_minor
Phase 3 (contract): stop writing balance, drop column
Rollback at Phase 1: old code reads balance column -> still there!
What about feature flags for irreversible actions?
Gate the irreversible action behind a feature flag so you can stop it instantly without a redeploy.
package com.bank.rollback;
public class FeatureFlagGating {
// Instead of: externalRail.wire(toBank, amount); // IRREVERSIBLE
// Use:
// if (featureFlags.isEnabled("new-wire-flow")) {
// outboxRepo.save(new OutboxEvent("WIRE", payload));
// // Wire happens only when relay picks up the outbox event
// }
// To stop: disable the flag -> relay stops processing wire events
// No redeploy needed, instant effect
public static void main(String[] args) {
System.out.println("Feature flag + outbox for irreversible actions:");
System.out.println(" 1. Gate new wire flow behind feature flag");
System.out.println(" 2. Wire goes through outbox relay");
System.out.println(" 3. Disable flag -> relay stops -> wires stop instantly");
System.out.println(" 4. No redeploy, no code rollback needed");
}
}
Expected output:
Feature flag + outbox for irreversible actions:
1. Gate new wire flow behind feature flag
2. Wire goes through outbox relay
3. Disable flag -> relay stops -> wires stop instantly
4. No redeploy, no code rollback needed
What is the WRONG way to handle rollback?
package com.bank.rollback;
public class WrongRollback {
// WRONG: stop writing the old balance column
// New code writes only balance_minor
// Old code reads balance -> NULL
// Rollback FAILS
// WRONG: publish only V2 events
// Old consumers cannot parse V2
// Rollback FAILS for consumers
// WRONG: wire money unconditionally in new flow
// Money is gone, rollback cannot un-send
// CORRECT: dual-write old AND new columns
// CORRECT: dual-publish V1 AND V2 events
// CORRECT: gate wire behind outbox + feature flag
public static void main(String[] args) {
System.out.println("WRONG: stop writing old column -> rollback breaks reads");
System.out.println("WRONG: publish only V2 events -> old consumers break");
System.out.println("WRONG: wire money unconditionally -> cannot un-send");
System.out.println();
System.out.println("CORRECT: dual-write, dual-publish, feature-flagged side effects");
}
}
Expected output:
WRONG: stop writing old column -> rollback breaks reads
WRONG: publish only V2 events -> old consumers break
WRONG: wire money unconditionally -> cannot un-send
CORRECT: dual-write, dual-publish, feature-flagged side effects
What should a senior engineer know?
In distributed systems you rarely truly roll back. You roll forward safely. The only way that is possible is to keep old and new state compatible throughout the change, which is a decision made while designing, not while firefighting.
Senior rollback checklist:
[x] Additive schema changes only (dual-write in expand phase)
[x] Dual-publish events in old and new format
[x] Feature flag on irreversible actions
[x] Outbox for side effects (wire, email, SMS)
[x] Canary release: exercise irreversible action on tiny fraction first
[x] Plan the backward path BEFORE shipping
[x] The safe rollback window closes at the first irreversible action
Visual Overview
Why Rollback Fails in Distributed Systems:
┌──────────────────────────────────────────────────┐
│ Deploy New Version (v2) │
│ │
│ 1. Writes new schema data ─┐ │
│ 2. Publishes V2 events ├─ STATE CHANGED │
│ 3. Wires money via NEFT │ │
│ ─┘ │
└──────────────────────┬───────────────────────────┘
│ Bug found! Roll back to v1
▼
┌──────────────────────────────────────────────────┐
│ Redeploy Old Version (v1) │
│ │
│ Old code faces: │
│ 1. Data in new format -> cannot read -> CRASH │
│ 2. V2 events in Kafka -> cannot parse -> FAIL │
│ 3. Money already wired -> cannot un-send │
│ │
│ CONCLUSION: code went back, state did NOT │
└──────────────────────────────────────────────────┘
Fix: Expand-and-Contract
┌────────┐ ┌──────────────┐ ┌──────────┐
│ Expand │──▶│ Migrate │──▶│ Contract │
│ Add new│ │ Consumers │ │ Remove │
│ Keep │ │ switch to │ │ old │
│ old │ │ new │ │ │
│ │ │ │ │ │
│ SAFE │ │ SAFE │ │ NOT SAFE │
│ TO │ │ TO ROLLBACK │ │ TO │
│ ROLL │ │ (old still │ │ ROLLBACK │
│ BACK │ │ works) │ │ │
└────────┘ └──────────────┘ └──────────┘
Key Takeaway
Rollback is harder in distributed systems because code rollback and state rollback are two different things that people conflate. Redeploying the previous binary is trivial, but it only restores the previous system if the state the old code reads is still state it understands. The moment the new version writes data in a new format, publishes events old consumers cannot parse, or performs an irreversible external action, the old binary faces a world it was never built for.
The three concrete ways this bites are schema-incompatible data, where the old code reads a NULL column the new code stopped writing; unrecallable events, where V2 events already committed to Kafka cannot be un-published; and irreversible side effects, where money wired through NEFT or emails sent cannot be un-done by any code change. The safe rollback window closes the instant the first of these occurs.
At National Bank of India, a naive change stopped writing the old balance column, wired money unconditionally, and published V2 events. When a bug was found, rolling back the code left old consumers unable to parse V2 events, the balance column reading NULL, and money already sent. The redesigned version dual-wrote both columns, dual-published events, and gated the wire behind a feature flag with an outbox. Disabling the flag stopped wires instantly without a redeploy.
Remember one sentence: code rollback is easy, state rollback is not: once new data, events, or side effects exist the old binary cannot restore the old system, so design reversibility into every change with expand-and-contract, dual publishing, and feature-flagged side effects before you ship.
---