Definition
- BACKWARD -- a consumer using the new schema can read data written with the previous schema; safe changes are deleting a field or adding a field with a default; upgrade consumers first.
- FORWARD -- a consumer using the old schema can read data written with the new schema; safe changes are adding a field or deleting a field that had a default; upgrade producers first.
- FULL -- both backward and forward hold simultaneously, so only the safest changes pass: adding or removing fields that have defaults.
- NONE -- disables compatibility checking entirely, allowing any change, which is dangerous in production.
- Transitive variants -- BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, and FULL_TRANSITIVE check against all previous versions, not just the latest, which matters when consumers may read very old data.
- Required field rule -- a field without a default is required; adding it breaks BACKWARD compatibility because old data has no value for it and no default to fall back on.
Imagine National Bank of India keeps changing the transfer form. BACKWARD compatibility means a clerk trained on the new form can still read all the old filled forms in the archive. FORWARD means a clerk trained on the old form can still read a new form someone just filled in. FULL means both clerks can read both forms. NONE means nobody checks and chaos is allowed. If the new form adds a mandatory currency box but every old archived form left it blank, the new clerk is stuck. The fix is to print INR as the default in that box, so when the box is blank the clerk assumes INR and keeps working.
Understanding the Concepts
What are Schema Registry compatibility modes?
The Schema Registry tracks schemas per subject, which is usually topicName-value. Each time you register a new schema version, the registry compares it against existing versions using the subject's compatibility mode and rejects the registration if the rule is violated. This check happens at registration time, which is far better than discovering the break at runtime in production.
package com.bank.schema;
// Setting compatibility mode via REST API
// PUT /config/transfers-value
// {"compatibility": "BACKWARD"}
public class CompatibilityConfig {
public static void main(String[] args) {
System.out.println("Compatibility modes:");
System.out.println(" BACKWARD -- new reader, old data");
System.out.println(" FORWARD -- old reader, new data");
System.out.println(" FULL -- both directions");
System.out.println(" NONE -- no checks");
System.out.println(" BACKWARD_TRANSITIVE -- against ALL versions");
System.out.println(" FORWARD_TRANSITIVE -- against ALL versions");
System.out.println(" FULL_TRANSITIVE -- against ALL versions");
}
}
Compatibility modes:
BACKWARD -- new reader, old data
FORWARD -- old reader, new data
FULL -- both directions
NONE -- no checks
BACKWARD_TRANSITIVE -- against ALL versions
FORWARD_TRANSITIVE -- against ALL versions
FULL_TRANSITIVE -- against ALL versions
Why do compatibility modes exist?
In a banking system with multiple teams, producers and consumers are deployed independently. The fraud team might release a consumer on Tuesday while the transfers team releases the producer on Thursday. Compatibility modes guarantee that these independent deployments do not break each other. Without them, a schema change by the producer could silently corrupt data for every consumer.
package com.bank.deploy;
public class IndependentDeployment {
public static void main(String[] args) {
System.out.println("BACKWARD deployment order:");
System.out.println(" 1. Deploy consumer with new schema (reads old + new)");
System.out.println(" 2. Deploy producer with new schema");
System.out.println();
System.out.println("FORWARD deployment order:");
System.out.println(" 1. Deploy producer with new schema");
System.out.println(" 2. Deploy consumer with new schema (old reads new)");
}
}
BACKWARD deployment order:
1. Deploy consumer with new schema (reads old + new)
2. Deploy producer with new schema
FORWARD deployment order:
1. Deploy producer with new schema
2. Deploy consumer with new schema (old reads new)
How does BACKWARD compatibility work?
BACKWARD is the default and most common mode in event systems. A consumer upgraded to the new schema can still read messages produced with the old schema. Safe backward changes are deleting a field and adding a field that has a default.
// Version 1 of Transfer schema
{
"type": "record",
"name": "Transfer",
"namespace": "com.bank.avro",
"fields": [
{"name": "accountId", "type": "string"},
{"name": "amount", "type": "string"},
{"name": "type", "type": "string"}
]
}
// Version 2 -- adds currency with a default (BACKWARD compatible)
{
"type": "record",
"name": "Transfer",
"namespace": "com.bank.avro",
"fields": [
{"name": "accountId", "type": "string"},
{"name": "amount", "type": "string"},
{"name": "type", "type": "string"},
{"name": "currency", "type": "string", "default": "INR"}
]
}
Why does adding a required field break compatibility?
In Avro, a field without a default is required. If you add a required field, a consumer reading an old message that was written before the field existed finds no value for it and no default to fall back on, so decoding fails. The registry rejects this registration under BACKWARD mode.
package com.bank.schema;
public class RequiredFieldBreak {
public static void main(String[] args) {
System.out.println("Scenario: adding 'currency' without a default");
System.out.println();
System.out.println("Old record written with v1:");
System.out.println(" {accountId: SAV2001, amount: 50000, type: DEBIT}");
System.out.println();
System.out.println("New consumer reads with v2 schema:");
System.out.println(" Expects: currency field");
System.out.println(" Found : no currency field, no default");
System.out.println(" Result : DECODE FAILS");
System.out.println();
System.out.println("Fix: add default to currency field");
System.out.println(" {name: currency, type: string, default: INR}");
System.out.println(" Old record decoded: currency = INR (from default)");
}
}
Scenario: adding 'currency' without a default
Old record written with v1:
{accountId: SAV2001, amount: 50000, type: DEBIT}
New consumer reads with v2 schema:
Expects: currency field
Found : no currency field, no default
Result : DECODE FAILS
Fix: add default to currency field
{name: currency, type: string, default: INR}
Old record decoded: currency = INR (from default)
How do you validate compatibility in CI?
The schema-registry-maven-plugin can test compatibility during the build, so a bad schema fails CI instead of failing a consumer in production.
<!-- pom.xml -- schema-registry-maven-plugin --syn-kw">>
syn-kw"><pluginsyn-kw">>
syn-kw"><groupIdsyn-kw">>io.confluentsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-schema-registry-maven-pluginsyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>7.5.0syn-kw"></versionsyn-kw">>
syn-kw"><configurationsyn-kw">>
syn-kw"><schemaRegistryUrlssyn-kw">>
syn-kw"><paramsyn-kw">>http://schema-registry:8081syn-kw"></paramsyn-kw">>
syn-kw"></schemaRegistryUrlssyn-kw">>
syn-kw"><subjectssyn-kw">>
syn-kw"><transfers-valuesyn-kw">>
src/main/avro/Transfer.avsc
syn-kw"></transfers-valuesyn-kw">>
syn-kw"></subjectssyn-kw">>
syn-kw"></configurationsyn-kw">>
syn-kw"><executionssyn-kw">>
syn-kw"><executionsyn-kw">>
syn-kw"><goalssyn-kw">>
syn-kw"><goalsyn-kw">>test-compatibilitysyn-kw"></goalsyn-kw">>
syn-kw"></goalssyn-kw">>
syn-kw"></executionsyn-kw">>
syn-kw"></executionssyn-kw">>
syn-kw"></pluginsyn-kw">>
# Run compatibility check in CI
mvn schema-registry:test-compatibility
What are the rules for each mode? (comparison)
| Change | BACKWARD | FORWARD | FULL | NONE |
|---|---|---|---|---|
| Add field with default | Safe | Safe | Safe | Safe |
| Add field without default | Breaks | Safe | Breaks | Safe |
| Delete field with default | Safe | Safe | Safe | Safe |
| Delete field without default | Safe | Breaks | Breaks | Safe |
| Rename field | Breaks | Breaks | Breaks | Safe |
package com.bank.schema;
public class CompatibilityRules {
public static void main(String[] args) {
System.out.println("Safe changes per mode:");
System.out.println(" BACKWARD : add with default, delete any");
System.out.println(" FORWARD : add any, delete with default");
System.out.println(" FULL : add with default, delete with default");
System.out.println(" NONE : anything goes (dangerous!)");
}
}
Safe changes per mode:
BACKWARD : add with default, delete any
FORWARD : add any, delete with default
FULL : add with default, delete with default
NONE : anything goes (dangerous!)
What are common mistakes?
package com.bank.mistake;
public class SchemaMistakes {
public static void main(String[] args) {
// WRONG: adding required field to an existing topic
System.out.println("WRONG: {name: currency, type: string}");
System.out.println(" Registry rejects under BACKWARD mode");
System.out.println();
// CORRECT: add field with a default
System.out.println("CORRECT: {name: currency, type: string, default: INR}");
System.out.println(" Registry accepts, old records decode with INR");
System.out.println();
// WRONG: setting NONE in production
System.out.println("WRONG: compatibility=NONE on transfers-value");
System.out.println(" Allows breaking changes without any check");
System.out.println();
// CORRECT: use BACKWARD or FULL
System.out.println("CORRECT: compatibility=BACKWARD on transfers-value");
System.out.println(" Registry enforces safe evolution");
}
}
WRONG: {name: currency, type: string}
Registry rejects under BACKWARD mode
CORRECT: {name: currency, type: string, default: INR}
Registry accepts, old records decode with INR
WRONG: compatibility=NONE on transfers-value
Allows breaking changes without any check
CORRECT: compatibility=BACKWARD on transfers-value
Registry enforces safe evolution
When should you use each mode?
- Use BACKWARD (the default) for most event topics where consumers are upgraded before producers.
- Use FORWARD when producers must be upgraded first and consumers catch up later.
- Use FULL for shared schemas across many teams where maximum safety is required.
- Use NONE only during early development on brand-new topics, and switch to BACKWARD before going live.
- Use transitive variants when consumers may read data many versions old, such as historical replay from the beginning of a topic.
What should a senior engineer know?
A senior engineer understands that compatibility modes turn schema evolution from a coordinated, risky, big-bang release into a safe, independent, incremental process. The registry check runs at registration time, ideally in CI through the schema-registry-maven-plugin, so a bad schema fails the build instead of failing a consumer in production. The deployment order depends on the mode: BACKWARD means consumers first, FORWARD means producers first. Transitive modes matter when consumers replay from the start of a topic and must read schemas that are several versions old.
Visual Overview
Schema Evolution with BACKWARD Compatibility
v1 schema v2 schema (adds currency with default)
┌────────────┐ ┌─────────────────────────┐
│ accountId │ │ accountId │
│ amount │ │ amount │
│ type │ │ type │
│ │ │ currency (default: INR) │
└────────────┘ └─────────────────────────┘
│ │
│ Old record │ New consumer reads old record
│ (no currency) │ currency = INR (from default)
▼ ▼
┌──────────────────────────────────────────┐
│ Schema Registry │
│ Subject: transfers-value │
│ Mode : BACKWARD │
│ │
│ v1 registered ✓ │
│ v2 registered ✓ (compatible) │
│ v2 without default ✗ (rejected!) │
└──────────────────────────────────────────┘
Key Takeaway
Schema Registry compatibility modes are the rules the registry enforces when a new schema version is registered, and they let producers and consumers evolve on their own schedules without breaking each other. BACKWARD, the default, means a consumer on the new schema can still read messages written with the old schema, so you upgrade consumers first. FORWARD means a consumer on the old schema can read messages written with the new schema, so you upgrade producers first. FULL requires both directions, and NONE turns the checks off.
The subtle point interviewers probe is about required fields. In Avro, a field without a default is required. If you add a required field, a consumer reading an older record that predates the field has no value and no default, so decoding fails and the registry rejects it. The fix is to give the new field a default. At National Bank of India, when Rajesh needed to add a currency field, the first attempt made it required and the registry rejected it immediately, which saved a production incident. He added a default of INR and it registered cleanly.
Pankaj always validates compatibility in the CI pipeline through the schema-registry-maven-plugin, so a bad schema fails the build instead of failing a consumer in production. The deployment order depends on the mode, and transitive variants matter when consumers replay from the beginning of a topic.
Remember one sentence: compatibility modes turn schema evolution from a risky big-bang release into a safe, independent process, and a new field is only safe if it has a default so old data can still be decoded.
---