Definition
- Breaking Change – a schema change where new events cannot be understood by consumers written against the old schema (removed field, changed type).
- Additive Evolution – only add new optional fields, never remove or rename existing ones; old consumers ignore new fields and keep working.
- Dual Publication – publish both v1 and v2 events during a controlled deprecation window so consumers migrate on their own schedules.
- Schema Registry – a central store that enforces compatibility rules, rejecting incompatible schema changes at publish time.
- Consumer Inventory – the set of all teams and services that consume the event, including undocumented ones.
- Organizational Migration – the communication, timeline, tracking, and deprecation process that gets fifteen teams across the line.
National Bank of India sends a standard "transaction completed" slip to fifteen departments: fraud, statements, rewards, tax, audit, and more. Changing the slip format overnight would leave every department that has not updated its process at the same minute unable to read it. So the bank either adds new fields without removing old ones, or prints both old and new slips for a few months while each department migrates on its own schedule.
Understanding the Concepts
Why can you not flip the schema in one deployment?
Fifteen consumers have fifteen independent deployment schedules, on-call rotations, and backlogs. Requiring simultaneous deployment is a distributed lockstep guaranteed to fail.
package com.bank.events;
public class SimultaneousDeployRisk {
public static void main(String[] args) {
String[] teams = {"fraud", "statements", "rewards", "tax", "audit",
"notifications", "analytics", "compliance", "risk", "marketing",
"mobile", "web", "partner-api", "reconciliation", "reporting"};
System.out.println("Teams that must deploy simultaneously: " + teams.length);
System.out.println("Team 3 is mid-incident, team 7 is on holiday, team 12 is frozen");
System.out.println("If even ONE is late → that consumer breaks");
System.out.println("Possibly fraud detection or audit → unacceptable for a bank");
}
}
Teams that must deploy simultaneously: 15 Team 3 is mid-incident, team 7 is on holiday, team 12 is frozen If even ONE is late → that consumer breaks Possibly fraud detection or audit → unacceptable for a bank
How does additive evolution work?
Only add new optional fields, never remove or rename existing ones. Old consumers ignore new fields and keep working.
package com.bank.events;
public class AdditiveEvolution {
// Original event: all consumers know these fields
record TransferCompletedV1(String transferId, long amountMinor) {}
// Additive: NEW fields added, OLD fields KEPT
record TransferCompletedAdditive(
String transferId,
long amountMinor, // KEPT
String currency, // NEW optional
String settlementRailV2 // NEW optional
) {}
public static void main(String[] args) {
System.out.println("Old consumers: read transferId and amountMinor, ignore new fields");
System.out.println("New consumers: read all fields including currency and settlementRailV2");
System.out.println("No coordinated deploy needed. Backward compatible.");
}
}
Old consumers: read transferId and amountMinor, ignore new fields
New consumers: read all fields including currency and settlementRailV2
No coordinated deploy needed. Backward compatible.
How does dual publication work for breaking changes?
Publish both v1 and v2 during a controlled deprecation window. Each team migrates on its own schedule.
package com.bank.events;
public class DualPublication {
static void publish(String transferId, long amountMinor, String currency) {
// Publish v1 for old consumers
System.out.println("Published: transfer.completed.v1 {transferId=" + transferId
+ ", amountMinor=" + amountMinor + "}");
// Publish v2 for new consumers
System.out.println("Published: transfer.completed.v2 {transferId=" + transferId
+ ", amountMinor=" + amountMinor + ", currency=" + currency + "}");
}
public static void main(String[] args) {
publish("TRF-001", 5000000, "INR");
System.out.println();
System.out.println("Old consumers read v1 topic. New consumers read v2 topic.");
System.out.println("After ALL consumers moved to v2 AND v1 has zero readers:");
System.out.println(" → Stop publishing v1.");
}
}
Published: transfer.completed.v1 {transferId=TRF-001, amountMinor=5000000}
Published: transfer.completed.v2 {transferId=TRF-001, amountMinor=5000000, currency=INR}
Old consumers read v1 topic. New consumers read v2 topic.
After ALL consumers moved to v2 AND v1 has zero readers:
→ Stop publishing v1.
What role does a schema registry play?
A schema registry enforces compatibility rules, rejecting incompatible changes at publish time before they can break consumers.
package com.bank.events;
public class SchemaRegistryCheck {
public static void main(String[] args) {
System.out.println("Schema Registry compatibility mode: BACKWARD");
System.out.println();
System.out.println("Attempt: remove 'amountMinor' field from TransferCompleted");
System.out.println("Registry response: REJECTED (breaking change, field removal)");
System.out.println();
System.out.println("Attempt: add optional 'currency' field");
System.out.println("Registry response: ACCEPTED (additive, backward compatible)");
}
}
Schema Registry compatibility mode: BACKWARD
Attempt: remove 'amountMinor' field from TransferCompleted
Registry response: REJECTED (breaking change, field removal)
Attempt: add optional 'currency' field
Registry response: ACCEPTED (additive, backward compatible)
Why is the schema only half the problem?
Fifteen teams depending on your event means the event is a contract. Changing a contract with fifteen counterparties is an organizational migration.
package com.bank.events;
public class OrganizationalMigration {
public static void main(String[] args) {
System.out.println("Technical half:");
System.out.println(" Schema registry, additive fields, dual publication");
System.out.println();
System.out.println("Organizational half:");
System.out.println(" 1. Discover ALL consumers (including undocumented ones)");
System.out.println(" 2. Publish deprecation timeline");
System.out.println(" 3. Provide migration guide and new schema");
System.out.println(" 4. Track each consumer's migration status");
System.out.println(" 5. Retire v1 ONLY when tracking confirms zero usage");
System.out.println();
System.out.println("Failure mode: 'We broke a consumer we did not know existed'");
}
}
Technical half:
Schema registry, additive fields, dual publication
Organizational half:
1. Discover ALL consumers (including undocumented ones)
2. Publish deprecation timeline
3. Provide migration guide and new schema
4. Track each consumer's migration status
5. Retire v1 ONLY when tracking confirms zero usage
Failure mode: 'We broke a consumer we did not know existed'
How do you compare the compatibility strategies?
| Strategy | When to Use | Coordination Needed | Risk |
|---|---|---|---|
| Additive evolution | Most changes (add fields) | None | Lowest |
| Dual publication | Genuinely breaking changes | Per-team migration tracking | Medium |
| Consumers before producer | New form is a superset | All consumers must be ready first | Higher |
| Big-bang flip | Never | Simultaneous deploy of 15 teams | Highest |
What are common mistakes?
Not discovering undocumented consumers before making the change.
package com.bank.events;
// WRONG: "We told the 15 known consumers. Change went live."
// Consumer #16 (undiscovered): data pipeline team reading the event directly from Kafka
// They break silently, audit data is now wrong
// CORRECT: discover ALL consumers from Kafka consumer group metadata,
// access logs, and schema registry subscriptions before any change
What are common interview follow-up questions?
- How do you discover undocumented consumers of a Kafka topic?
- What is the difference between backward and forward compatibility in Avro?
- How do you track migration progress across fifteen teams?
- When can you safely stop publishing the old event version?
Visual Overview
Event Schema Migration Strategy
───────────────────────────────
Additive (preferred):
Producer publishes:
{transferId, amountMinor, currency(NEW)} ← v1+ fields
Old consumers: read transferId, amountMinor ← ignore currency
New consumers: read all fields ← use currency
No coordination needed.
Dual Publication (breaking change):
Producer publishes:
Topic v1: {transferId, amountMinor} ← old schema
Topic v2: {transferId, amountMinor, currency} ← new schema
┌─────────────────────────────────────────────┐
│ Deprecation window: 3 months │
│ Each team migrates v1 → v2 on own schedule │
│ Track: 12/15 migrated, 3 remaining │
│ Retire v1 ONLY when 15/15 and zero readers │
└─────────────────────────────────────────────┘
Key Takeaway
A breaking event-schema change with fifteen consumers cannot be flipped in one deployment because fifteen independent teams have fifteen deployment schedules. The whole strategy is about buying time so the producer's change and each consumer's change happen on independent schedules.
The safest technique is additive evolution: add new optional fields, never remove existing ones, so old consumers keep working. For genuinely breaking changes, dual-publish v1 and v2 during a controlled deprecation window and let each team migrate on its own schedule. A schema registry with backward-compatibility enforcement rejects incompatible changes before they can break consumers.
The part that matters most is that the technical schema is only half the problem. The event is a contract with fifteen counterparties, and changing it requires discovering all consumers (including undocumented ones), publishing a deprecation timeline, tracking migration status, and retiring the old version only when confirmed at zero usage.
Remember one sentence: never force fifteen teams to deploy on one clock; evolve additively or dual-publish with a tracked deprecation, and treat the change as an organizational migration.
---