Definition
- Schema Registry – a centralized service that stores and versions the schemas describing the structure of messages on Kafka topics, and enforces compatibility rules when schemas evolve.
- Avro – a compact binary serialization format that uses schemas with named fields and types, commonly paired with Schema Registry for Kafka messages.
- Protobuf – Google's binary serialization format, another option for schema-enforced Kafka messages alongside Avro and JSON Schema.
- Schema id – a numeric identifier assigned by the registry when a schema is registered; the producer embeds this id (typically 4 bytes) at the start of each message instead of the full schema.
- Compatibility mode – the rule enforced when a new schema version is registered: backward (new consumers read old data), forward (old consumers read new data), full (both), or none.
- Schema evolution – the process of changing a schema over time (adding fields, removing optional fields) while maintaining compatibility with existing producers and consumers.
Imagine National Bank of India sends transfer forms between departments. Sending plain JSON is like letting every clerk write the form freehand: one writes "amt", another writes "amount", someone forgets the currency, and the receiving department silently misreads it. A Schema Registry is like the bank publishing an official, numbered form template that everyone must use, with rules that any new version must still be readable by departments using the old version. Every form references the template number instead of reprinting the whole template, saving paper, and the mailroom refuses any form that does not match an approved template.
Understanding the Concepts
What is the problem with raw JSON?
The central problem with sending raw JSON is that there is no enforced contract. JSON is self-describing but schemaless from Kafka's point of view: the producer can add, rename, remove, or retype fields at any time, and Kafka happily carries whatever bytes it is given. A consumer expecting a numeric amount may suddenly receive a string, a renamed field, or a missing field, and it breaks at runtime in production.
package com.bank.json;
public class RawJsonProblem {
public static void main(String[] args) {
// Producer v1: {"accountId":"SAV2001","amount":50000}
// Producer v2: {"account_id":"SAV2001","amt":"50000.00"}
// Consumer expects: record.get("accountId") -> null!
// record.get("amount") -> null!
System.out.println("Raw JSON Problems:");
System.out.println(" 1. Producer renames 'accountId' to 'account_id'");
System.out.println(" Consumer gets null -> NullPointerException");
System.out.println();
System.out.println(" 2. Producer changes 'amount' from int to string");
System.out.println(" Consumer casts to Integer -> ClassCastException");
System.out.println();
System.out.println(" 3. Producer removes 'currency' field");
System.out.println(" Consumer assumes INR -> processes USD as INR");
System.out.println();
System.out.println(" All discovered at RUNTIME in PRODUCTION");
}
}
Expected output:
Raw JSON: field renamed -> NPE, type changed -> ClassCastException
All discovered at runtime in production. Dangerous for banking.
How does a Schema Registry solve this?
A Schema Registry makes the schema a first-class, versioned artifact. When a producer sends a record, the serializer registers or looks up the schema in the registry and gets back a numeric schema id. The message on the wire is the schema id (4 bytes) plus the compactly encoded data. The consumer's deserializer reads the id, fetches the exact schema, and decodes reliably.
package com.bank.config;
import java.util.Properties;
public class SchemaRegistryProducerConfig {
public static Properties createConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");
System.out.println("Producer configured with Schema Registry:");
System.out.println(" Serializer: KafkaAvroSerializer");
System.out.println(" Registry: http://localhost:8081");
System.out.println(" Schema registered/cached automatically");
return props;
}
public static void main(String[] args) {
createConfig();
}
}
Expected output:
Producer: KafkaAvroSerializer + schema.registry.url
Schema auto-registered, messages carry compact schema id + binary data
How does schema evolution with compatibility work?
The registry enforces a configured compatibility mode when a new schema version is registered. If a producer team tries to register a change that breaks the rule, the registry rejects the registration at deploy time, not at runtime.
| Compatibility Mode | Rule | Example |
|---|---|---|
| BACKWARD | New consumers can read old data | Add field with default |
| FORWARD | Old consumers can read new data | Remove optional field |
| FULL | Both backward and forward | Add field with default only |
| NONE | No compatibility check | Any change allowed |
package com.bank.schema;
public class SchemaEvolutionExample {
public static void main(String[] args) {
System.out.println("Schema Evolution (BACKWARD compatibility):");
System.out.println();
System.out.println("Schema v1:");
System.out.println(" {accountId: string, amount: double}");
System.out.println();
System.out.println("Schema v2 (add field with default):");
System.out.println(" {accountId: string, amount: double,");
System.out.println(" currency: string = 'INR'}");
System.out.println(" -> ALLOWED: new consumers read old data (default INR)");
System.out.println();
System.out.println("Schema v3 (remove required field):");
System.out.println(" {amount: double, currency: string}");
System.out.println(" -> REJECTED: old consumers expect 'accountId'");
System.out.println(" Error at DEPLOY time, not in production!");
}
}
Expected output:
v2 (add field with default): ALLOWED
v3 (remove required field): REJECTED at deploy time
Breaking changes caught before production.
What is the payload efficiency benefit?
With Avro or Protobuf plus a schema id, the message does not repeat field names in every record. It carries a 4-byte schema id plus compact binary data. JSON, by contrast, repeats every key name in every message. Over millions of transaction events, this significantly reduces network bandwidth and storage.
package com.bank.efficiency;
public class PayloadSizeComparison {
public static void main(String[] args) {
// JSON: {"accountId":"SAV2001","amount":50000.00,"currency":"INR"}
String json = "{\"accountId\":\"SAV2001\",\"amount\":50000.00,\"currency\":\"INR\"}";
// Avro: 4-byte schema id + binary encoded values
// Approximate: 4 + 7 + 8 + 3 = 22 bytes
int avroSize = 22;
System.out.println("JSON payload: " + json.length() + " bytes");
System.out.println("Avro payload: ~" + avroSize + " bytes");
System.out.println("Savings: ~" + (json.length() - avroSize) + " bytes/message");
System.out.println();
long messagesPerDay = 10_000_000;
long savedPerDay = messagesPerDay * (json.length() - avroSize);
System.out.println("At 10M messages/day:");
System.out.println(" Savings: ~" + (savedPerDay / (1024 * 1024)) + " MB/day");
}
}
Expected output:
JSON payload: 57 bytes
Avro payload: ~22 bytes
Savings: ~35 bytes/message
At 10M messages/day:
Savings: ~333 MB/day
How do you use Schema Registry with Spring Kafka?
In Spring Boot, you configure the Avro serializer and the registry URL. The producer automatically registers schemas and embeds the schema id. The consumer fetches the schema by id and deserializes reliably.
package com.bank.config;
public class SpringKafkaSchemaConfig {
public static void main(String[] args) {
System.out.println("application.yml for Schema Registry:");
System.out.println();
System.out.println("spring:");
System.out.println(" kafka:");
System.out.println(" producer:");
System.out.println(" value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer");
System.out.println(" consumer:");
System.out.println(" value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer");
System.out.println(" properties:");
System.out.println(" schema.registry.url: http://localhost:8081");
System.out.println(" specific.avro.reader: true");
System.out.println();
System.out.println("Producer sends Avro -> schema id embedded in message");
System.out.println("Consumer reads schema id -> fetches schema -> deserializes");
}
}
Expected output:
Spring config: KafkaAvroSerializer + KafkaAvroDeserializer
schema.registry.url for both producer and consumer
What are the trade-offs?
The registry is another service to run and make highly available, and it adds a small lookup on first use of each schema (cached afterward). JSON is simpler for small systems, human-readable, and easy to debug by eye. You can even use JSON Schema inside the registry to keep readability while gaining enforcement.
| Factor | Raw JSON | Schema Registry (Avro) |
|---|---|---|
| Contract enforcement | None | Strict, versioned |
| Payload size | Large (field names repeated) | Small (binary + schema id) |
| Schema evolution | Uncontrolled | Compatibility enforced |
| Human readability | Easy (plain text) | Hard (binary, need tools) |
| Operational cost | None | Registry service to manage |
| Debug ease | Read message directly | Need Avro tools or registry UI |
| Best for | Small systems, prototyping | Multi-team, financial systems |
package com.bank.tradeoffs;
public class SchemaRegistryTradeoffs {
public static void main(String[] args) {
System.out.println("Schema Registry Trade-offs:");
System.out.println();
System.out.println("Benefits:");
System.out.println(" + Enforced, versioned contract");
System.out.println(" + Smaller payloads (binary)");
System.out.println(" + Breaking changes caught at deploy time");
System.out.println(" + Safe schema evolution across teams");
System.out.println();
System.out.println("Costs:");
System.out.println(" - Another service to operate and keep HA");
System.out.println(" - Small latency on first schema lookup");
System.out.println(" - Binary messages harder to read by eye");
System.out.println(" - Team must learn Avro/Protobuf schema syntax");
}
}
Benefits: enforced contracts, smaller payloads, safe evolution
Costs: extra service, binary messages, learning curve
What are common mistakes with Schema Registry?
The most common mistake is disabling compatibility checks (NONE mode) to "move fast", which defeats the purpose of the registry. Another mistake is not running the registry in HA mode, making it a single point of failure.
package com.bank.mistakes;
// WRONG: Compatibility mode set to NONE.
// Any schema change is allowed, including breaking ones.
// Producer renames 'amount' to 'amt' -> consumers break.
public class WrongCompatibilityMode {
public static void main(String[] args) {
System.out.println("WRONG: compatibility = NONE");
System.out.println(" Producer registers breaking change -> allowed");
System.out.println(" Consumer crashes in production -> same as raw JSON");
System.out.println();
System.out.println("CORRECT: compatibility = BACKWARD (or FULL)");
System.out.println(" Producer tries breaking change -> REJECTED");
System.out.println(" Breaking change caught at deploy time");
System.out.println(" Consumers never see incompatible data");
}
}
WRONG: NONE mode = no protection = same as raw JSON
CORRECT: BACKWARD or FULL = breaking changes rejected at deploy time
When should you use Schema Registry vs raw JSON?
For a banking event backbone with many producers and consumers evolving independently, a Schema Registry with a strict compatibility mode is the professional default. For a tiny internal tool or a prototype, raw JSON is acceptable.
package com.bank.decision;
public class SchemaRegistryDecision {
public static void main(String[] args) {
System.out.println("Decision Guide:");
System.out.println();
System.out.println("Use Schema Registry:");
System.out.println(" - Multi-team banking event backbone");
System.out.println(" - Financial data (amount, currency sensitive)");
System.out.println(" - More than 3 producer/consumer teams");
System.out.println(" - Need schema evolution without downtime");
System.out.println(" - High-volume topics (payload savings matter)");
System.out.println();
System.out.println("Raw JSON acceptable:");
System.out.println(" - Internal tool with 1 producer, 1 consumer");
System.out.println(" - Prototype or POC");
System.out.println(" - Non-critical data where loss is tolerable");
}
}
Expected output:
Schema Registry: multi-team, financial, high-volume
Raw JSON: single-team, prototype, non-critical
What should a senior engineer know?
A senior engineer treats the message format as a governed, versioned API contract rather than an informal convention. After a production incident at National Bank of India where a producer team renamed a field and several consumers broke, the team adopted a Schema Registry with backward compatibility enforced. Now every transfer topic has a registered schema, and an attempt to remove a required field is rejected before it ships.
package com.bank.senior;
public class SchemaRegistrySeniorInsights {
public static void main(String[] args) {
System.out.println("Senior Schema Registry Insights:");
System.out.println("1. Schema = API contract (versioned, governed)");
System.out.println("2. BACKWARD compatibility for most banking topics");
System.out.println("3. Run registry in HA mode (3-node minimum)");
System.out.println("4. Register schemas in CI/CD pipeline");
System.out.println("5. Use subject naming strategy per topic");
System.out.println("6. Monitor schema registration failures");
System.out.println("7. Consider JSON Schema if team prefers readability");
System.out.println("8. Cost of unenforced contract >> cost of registry");
}
}
Expected output:
Senior insights: schema = API contract, BACKWARD compatibility,
run in HA, register in CI/CD, cost of unenforced contract >> registry cost
Visual Overview
Schema Registry Flow
=====================
Producer Schema Registry Consumer
│ │ │
│ 1. Register schema │ │
│ (or lookup cached id) │ │
├────────────────────────▶│ │
│ │ │
│ 2. Schema id = 42 │ │
│◀────────────────────────│ │
│ │ │
│ 3. Send message: │ │
│ [schema_id=42][binary_data] │
│─────────────────────────────────────────────────│
│ │ │
│ │ 4. Fetch schema 42 │
│ │◀──────────────────────│
│ │ │
│ │ 5. Schema returned │
│ │──────────────────────▶│
│ │ │
│ │ 6. Deserialize using │
│ │ schema 42 │
│ │ (cached for next) │
Compatibility Check (at registration):
┌────────────────────────────────────────────┐
│ New schema v2: add 'currency' with default │
│ BACKWARD check: can v2 consumer read v1? │
│ Result: YES (default value fills gap) │
│ Schema v2 REGISTERED (id=43) │
└────────────────────────────────────────────┘
┌────────────────────────────────────────────┐
│ New schema v3: remove 'accountId' │
│ BACKWARD check: can v3 consumer read v2? │
│ Result: NO (v2 has accountId, v3 drops it) │
│ Schema v3 REJECTED (breaking change!) │
└────────────────────────────────────────────┘
Key Takeaway
A Schema Registry turns the Kafka message format into a governed, versioned API contract that prevents breaking changes from reaching production. Raw JSON has no enforced schema: a producer can rename, remove, or retype a field at any time, and Kafka carries whatever bytes it receives. A consumer expecting a numeric amount may suddenly get a string or a missing field, and it breaks at runtime. In banking, a silently misread transfer amount is a serious incident.
The Schema Registry makes the schema a first-class artifact. The producer serializes with a registered schema (typically Avro or Protobuf), and the message carries a compact schema id plus binary data instead of repeating field names. The consumer reads the id, fetches the schema, and decodes reliably. Compatibility modes (backward, forward, full) enforce that new schema versions do not break existing producers or consumers, and a breaking change is rejected at registration time, not discovered by a crashing consumer in production.
The trade-off is operational: the registry is another service to run and keep highly available, and binary messages are harder to read by eye than JSON. But for a banking event backbone with many teams evolving independently, the cost of an unenforced contract far outweighs running a registry. After National Bank of India had a production incident where a renamed field broke several consumers, every financial topic now has a registered schema with backward compatibility enforced.
Remember one sentence: a Schema Registry enforces versioned, compatible message contracts at registration time, catching breaking changes before production and shrinking payloads compared to raw JSON.
---