Definition
- Additive change – Add a new optional field with a default; old consumers ignore it, new consumers use it.
- Expand-and-contract – Add the new field (expand), let consumers migrate, then remove the old field (contract).
- Provider contract tests – Each consumer publishes the shape it relies on; the provider's build runs those contracts and fails on breaks.
- Semantic break – The structure is unchanged but the meaning of a field changes; invisible to structural tests.
- Schema registry – A central store (Confluent, Apicurio) that validates event schemas against compatibility rules before registration.
Imagine National Bank of India sends a standard transfer notice that twenty departments read. If you suddenly change what a field means, every department silently misreads it. So the bank only adds optional new lines, keeps existing meanings fixed, announces well in advance before removing anything, and keeps a rulebook of the exact format so old archived notices can still be understood years later.
Understanding the Concepts
What are backward and forward compatible changes?
Backward compatible means a new producer's output is readable by old consumers. Forward compatible means an old producer's output is readable by new consumers.
package com.bank.contract;
public class CompatibilityTypes {
public static void main(String[] args) {
System.out.println("Backward compatible: new producer -> old consumer can read");
System.out.println(" Adding optional field with default: old consumer ignores it");
System.out.println();
System.out.println("Forward compatible: old producer -> new consumer can read");
System.out.println(" New consumer sees default when field is absent");
System.out.println();
System.out.println("BREAKING changes (avoid):");
System.out.println(" Removing a field");
System.out.println(" Renaming a field");
System.out.println(" Changing a field's type");
System.out.println(" Making an optional field required");
}
}
Expected output:
Backward compatible: new producer -> old consumer can read
Adding optional field with default: old consumer ignores it
Forward compatible: old producer -> new consumer can read
New consumer sees default when field is absent
BREAKING changes (avoid):
Removing, renaming, retyping, or requiring a field
How does expand-and-contract work?
Add the new field first, let consumers migrate, then remove the old field. Never a flag-day migration.
package com.bank.contract;
public class ExpandAndContract {
public static void main(String[] args) {
System.out.println("Phase 1 (EXPAND): add new field, keep old");
System.out.println(" Event: {amount: 50000, amount_minor: 5000000}");
System.out.println(" Old consumers read 'amount' (still there)");
System.out.println(" New consumers read 'amount_minor' (new)");
System.out.println();
System.out.println("Phase 2 (MIGRATE): consumers switch to new field");
System.out.println(" Verify: no consumer reads 'amount' anymore");
System.out.println();
System.out.println("Phase 3 (CONTRACT): stop writing old field");
System.out.println(" Event: {amount_minor: 5000000}");
System.out.println(" Only safe AFTER all consumers migrated");
}
}
Expected output:
Phase 1 (EXPAND): add new field, keep old
Phase 2 (MIGRATE): consumers switch to new field
Phase 3 (CONTRACT): stop writing old field
What are provider contract tests?
Each consumer publishes the shape it relies on. The provider's build runs those contracts.
package com.bank.contract;
public class ContractTests {
public static void main(String[] args) {
System.out.println("Consumer-driven contract testing:");
System.out.println(" Account-service consumer expects: {accountId, balance, holderName}");
System.out.println(" Statement-service consumer expects: {accountId, balance, txnHistory}");
System.out.println();
System.out.println(" Provider's CI pipeline:");
System.out.println(" 1. Run account-service contract against current output");
System.out.println(" 2. Run statement-service contract against current output");
System.out.println(" 3. If provider removed 'balance' -> BOTH contracts FAIL");
System.out.println(" 4. Build blocked before deployment");
System.out.println();
System.out.println(" Turns 'we think it is safe' into 'the build proves it is safe'");
}
}
Expected output:
Consumer-driven contract testing:
Provider removed 'balance' -> BOTH contracts FAIL -> build blocked
Turns 'we think it is safe' into 'the build proves it is safe'
What is a semantic break and why is it dangerous?
The structure is unchanged but the meaning of a field changes. Every structural test passes while every consumer is silently wrong.
package com.bank.contract;
import java.math.BigDecimal;
public class SemanticBreak {
// Before: amount field is in RUPEES (50000.00)
// After: amount field is in PAISE (5000000)
// JSON structure: IDENTICAL
// Schema check: PASSES
// Contract test (structural): PASSES
// Consumer behavior: WRONG by factor of 100!
public static void main(String[] args) {
BigDecimal amountInRupees = new BigDecimal("50000.00");
BigDecimal amountInPaise = new BigDecimal("5000000");
System.out.println("SEMANTIC BREAK:");
System.out.println(" Before: amount = " + amountInRupees + " (rupees)");
System.out.println(" After: amount = " + amountInPaise + " (paise)");
System.out.println(" JSON looks the same: {\"amount\": <number>}");
System.out.println(" Every structural test PASSES");
System.out.println(" Every consumer reads 100x wrong value!");
System.out.println();
System.out.println("Fix: NEVER repurpose a field. Add a new field instead.");
System.out.println(" {amount: 50000.00, amount_minor: 5000000, currency: INR}");
}
}
Expected output:
SEMANTIC BREAK:
Before: amount = 50000.00 (rupees)
After: amount = 5000000 (paise)
JSON looks the same
Every structural test PASSES
Every consumer reads 100x wrong value!
Fix: NEVER repurpose a field. Add a new field instead.
Why treat Kafka events as durable contracts?
A REST response is consumed immediately. A Kafka event can be replayed months later. Old events must still be parseable.
package com.bank.contract;
public class KafkaDurableContract {
public static void main(String[] args) {
System.out.println("Kafka events are DURABLE:");
System.out.println(" Event written January 2024");
System.out.println(" Replayed in March 2024 by a new consumer");
System.out.println(" Consumer must still parse the old format!");
System.out.println();
System.out.println("Schema registry (Confluent/Apicurio):");
System.out.println(" Compatibility mode: BACKWARD");
System.out.println(" New schema registered -> registry checks compatibility");
System.out.println(" Breaking schema -> REJECTED at registration time");
}
}
Expected output:
Kafka events are DURABLE:
Event written January 2024
Replayed in March 2024 by a new consumer
Consumer must still parse the old format!
Schema registry (Confluent/Apicurio):
Compatibility mode: BACKWARD
Breaking schema -> REJECTED at registration time
Why never publish internal JPA entities to Kafka?
An innocent column rename silently changes the public contract.
package com.bank.contract;
public class NeverPublishInternalEntity {
// WRONG: publish JPA entity directly
// @Entity Account { String holderName; BigDecimal balance; }
// kafkaTemplate.send("account-events", objectMapper.writeValueAsString(account));
// Later: refactor holderName -> customerName -> BREAKS every consumer
// CORRECT: explicit event class
// record AccountCreatedEvent(String accountId, String holderName, BigDecimal balance) {}
// Map entity -> event deliberately
// Event class is governed as a public API
public static void main(String[] args) {
System.out.println("WRONG: publish JPA entity -> column rename breaks consumers");
System.out.println("CORRECT: map to explicit event class -> governed API");
}
}
Expected output:
WRONG: publish JPA entity -> column rename breaks consumers
CORRECT: map to explicit event class -> governed API
How do you handle truly breaking changes?
Version the event type as a last resort. Like API versions, every live version is ongoing cost.
package com.bank.contract;
public class EventVersioning {
public static void main(String[] args) {
System.out.println("Truly breaking change that cannot be made additive:");
System.out.println(" 1. Create new event type: TransferCompletedV2");
System.out.println(" 2. Publish BOTH V1 and V2 (dual-publish)");
System.out.println(" 3. Consumers migrate to V2 on their schedule");
System.out.println(" 4. Stop publishing V1 after all consumers migrated");
System.out.println();
System.out.println("Cost: every live version is ongoing maintenance");
System.out.println(" -> Version only as a last resort");
}
}
Expected output:
Truly breaking change:
1. Create TransferCompletedV2
2. Dual-publish V1 and V2
3. Consumers migrate
4. Stop V1
Cost: every live version is ongoing maintenance
What should a senior engineer know?
Keep events small and intention-revealing. Document contracts like code. Prefer immutable, append-only event design.
Senior guidelines:
[x] Additive changes only (new optional fields with defaults)
[x] Deprecate before delete with concrete sunset date
[x] Provider contract tests in CI
[x] Never repurpose a field (semantic break defense)
[x] Schema registry with backward compatibility mode
[x] Explicit event classes, never internal entities
[x] Version event types only as last resort
[x] Encode units explicitly: {amount: 50000, currency: INR, scale: 2}
Visual Overview
Contract Evolution Flow:
Phase 1: EXPAND (add new, keep old)
Event: {amount: 50000.00, amount_minor: 5000000}
│ │
Old consumer New consumer
reads 'amount' reads 'amount_minor'
(still works) (new field)
Phase 2: MIGRATE (consumers switch)
All consumers now read 'amount_minor'
'amount' field still published but unused
Phase 3: CONTRACT (remove old)
Event: {amount_minor: 5000000}
Safe: no consumer reads 'amount' anymore
Schema Registry Guard:
┌──────────┐ register ┌──────────────┐
│ Producer │───────────▶│ Schema │ BACKWARD mode
│ (new │ │ Registry │ Breaking? -> REJECT
│ schema) │ │ │ Compatible? -> OK
└──────────┘ └──────────────┘
Key Takeaway
Safely evolving a contract that many services depend on is about changing a shared schema without forcing everyone to deploy on the same day. Additive changes, like adding an optional field with a default, preserve compatibility because old consumers ignore what they do not know and new consumers fall back to the default. Removing, renaming, or retyping a field is breaking and needs a managed migration via expand-and-contract.
Provider contract tests catch breaks at build time by running each consumer's expected shape against the provider's current output. Semantic breaks are the most dangerous: changing a field's meaning without changing its structure passes every structural test while silently corrupting every consumer. The defense is never repurposing a field and encoding units explicitly.
For Kafka events, which can be replayed months later, Pankaj at National Bank of India keeps the schema in a registry with backward compatibility mode and maps internal entities to explicit, governed event classes so a column rename never silently breaks the public contract.
Remember one sentence: only add optional fields, never change existing meanings, deprecate before you delete, verify with contract tests, and treat every Kafka event as a stable schema in a registry rather than your internal entity.
---