Definition
- Avro – a row-based binary format that stores each record's fields together and embeds the writer's schema in the file header, excelling at write-heavy workloads and schema evolution.
- Parquet – a columnar format storing each column's values together, enabling column pruning and predicate pushdown, the standard for read-heavy analytics.
- ORC – a columnar format with built-in lightweight indexes and first-class Hive ACID transaction support for row-level updates.
- Schema evolution – Avro's first-class feature for adding/removing fields without breaking consumers.
- Column pruning – Parquet's ability to read only the columns a query needs.
- Two-stage pipeline – land raw as Avro, curate as Parquet or ORC for analytics.
At the National Bank of India imagine three ways to store transaction records. Avro is like a stack of complete forms filed one after another, each form carrying a key explaining its fields: fast to add and easy to adapt when the form changes, ideal for the incoming mail room. Parquet is like a filing system that keeps all amounts in one drawer, all dates in another: so when analytics asks only for amounts, they open just one drawer. ORC is a similar drawer system but built into the Hive records room with extra index cards and the ability to amend filed records (ACID).
Understanding the Concepts
How does Avro work?
Avro stores data row by row, keeping all of one record's fields together, and writes the full schema in the file header in JSON. Writing a record means appending it whole, which is efficient for streaming ingestion. The embedded schema plus well-defined compatibility rules make schema evolution a first-class feature.
package com.bank.formats;
public class AvroOverview {
public static void main(String[] args) {
System.out.println("Avro layout (row-based):");
System.out.println(" Header: {schema JSON}");
System.out.println(" Record 1: [T1|SAV2001|15000|INR]");
System.out.println(" Record 2: [T2|CUR3002|25000|INR]");
System.out.println();
System.out.println("Strengths:");
System.out.println(" - Fast writes (append whole record)");
System.out.println(" - Schema evolution (add/remove fields)");
System.out.println(" - Pairs with Schema Registry for Kafka");
System.out.println(" - Compact binary encoding");
}
}
Avro layout (row-based):
Header: {schema JSON}
Record 1: [T1|SAV2001|15000|INR]
Record 2: [T2|CUR3002|25000|INR]
Strengths:
- Fast writes (append whole record)
- Schema evolution (add/remove fields)
- Pairs with Schema Registry for Kafka
- Compact binary encoding
How does Parquet work?
Parquet stores data column by column: all values of the amount column together, then all of date, organized into row groups and column chunks. This layout makes column pruning and predicate pushdown possible, and same-type values compress far better.
package com.bank.formats;
public class ParquetOverview {
public static void main(String[] args) {
System.out.println("Parquet layout (columnar):");
System.out.println(" txn_id column: [T1, T2, T3, ...]");
System.out.println(" amount column: [15000, 25000, ...]");
System.out.println(" date column: [Oct-29, Oct-29, ...]");
System.out.println();
System.out.println("Strengths:");
System.out.println(" - Column pruning (read only needed cols)");
System.out.println(" - Predicate pushdown (min/max skip)");
System.out.println(" - Strong compression (same-type values)");
System.out.println(" - Default for Spark, Trino, cloud lakes");
}
}
Parquet layout (columnar):
txn_id column: [T1, T2, T3, ...]
amount column: [15000, 25000, ...]
date column: [Oct-29, Oct-29, ...]
Strengths:
- Column pruning (read only needed cols)
- Predicate pushdown (min/max skip)
- Strong compression (same-type values)
- Default for Spark, Trino, cloud lakes
How does ORC work?
ORC is also columnar with the same analytical advantages, but it adds richer built-in indexes and statistics at file, stripe, and row-group levels, including optional bloom filters. Its standout feature is first-class Hive ACID transaction support.
package com.bank.formats;
public class OrcOverview {
public static void main(String[] args) {
System.out.println("ORC layout (columnar, Hive-optimized):");
System.out.println(" Similar to Parquet: column-oriented");
System.out.println();
System.out.println("ORC extras vs Parquet:");
System.out.println(" - Richer built-in indexes and stats");
System.out.println(" - Built-in bloom filters");
System.out.println(" - First-class Hive ACID support");
System.out.println(" - Row-level UPDATE/DELETE in Hive");
}
}
ORC layout (columnar, Hive-optimized):
Similar to Parquet: column-oriented
ORC extras vs Parquet:
- Richer built-in indexes and stats
- Built-in bloom filters
- First-class Hive ACID support
- Row-level UPDATE/DELETE in Hive
How do the three formats compare?
| Aspect | Avro | Parquet | ORC |
|---|---|---|---|
| Layout | Row-based | Columnar | Columnar |
| Schema | In file header (JSON) | In file footer | In file tail |
| Schema evolution | First-class | Supported | Supported |
| Column pruning | No (reads all cols) | Yes | Yes |
| Predicate pushdown | No | Yes (min/max) | Yes (indexes) |
| Compression | Good | Very good | Very good |
| ACID | No | No (needs Delta) | Yes (Hive) |
| Best for | Writes, Kafka, ingestion | Reads, analytics, Spark | Hive, ACID analytics |
package com.bank.formats;
public class FormatComparison {
public static void main(String[] args) {
System.out.println("Avro: write-heavy, schema evolution, Kafka");
System.out.println("Parquet: read-heavy, analytics, Spark/Trino");
System.out.println("ORC: Hive ecosystem, ACID updates");
}
}
Avro: write-heavy, schema evolution, Kafka
Parquet: read-heavy, analytics, Spark/Trino
ORC: Hive ecosystem, ACID updates
Why not analyze directly on Avro?
A columnar scan touching a few columns of a wide table reads a tiny fraction of the bytes, while Avro must read whole rows. On a 100-column table, Parquet reads only the 2 needed columns; Avro reads all 100.
package com.bank.formats;
public class WhyNotAvroForAnalytics {
public static void main(String[] args) {
System.out.println("SUM(amount) on 100-column table:");
System.out.println();
System.out.println("Avro: reads ALL 100 columns per row.");
System.out.println(" I/O: 10 GB");
System.out.println();
System.out.println("Parquet: reads ONLY amount column.");
System.out.println(" I/O: 100 MB");
System.out.println();
System.out.println("100x less I/O with Parquet.");
}
}
SUM(amount) on 100-column table:
Avro: reads ALL 100 columns per row.
I/O: 10 GB
Parquet: reads ONLY amount column.
I/O: 100 MB
100x less I/O with Parquet.
Why not stream events into Parquet?
Columnar formats want large batched writes. Streaming individual events into Parquet produces a small-files problem: thousands of tiny files that are inefficient to read. Use Avro for streaming and convert to Parquet in a batch compaction step.
package com.bank.formats;
public class WhyNotParquetForStreaming {
public static void main(String[] args) {
System.out.println("Streaming 1 event at a time into Parquet:");
System.out.println(" Creates 1 tiny file per event.");
System.out.println(" 1M events = 1M files.");
System.out.println(" Reading: slow (file open overhead).");
System.out.println();
System.out.println("Solution: two-stage pipeline");
System.out.println(" 1. Stream events -> Avro (row-friendly)");
System.out.println(" 2. Batch compact -> Parquet (col-friendly)");
}
}
Streaming 1 event at a time into Parquet:
Creates 1 tiny file per event.
1M events = 1M files.
Reading: slow (file open overhead).
Solution: two-stage pipeline
1. Stream events -> Avro (row-friendly)
2. Batch compact -> Parquet (col-friendly)
When do you use each format?
Use Avro for the write and ingestion path: streaming events, Kafka messages, and anywhere the schema evolves. Use Parquet for the analytical read path: curated tables scanned by reports. Use ORC in the Hive ecosystem when ACID updates are needed. Modern lakehouses (Delta, Iceberg) layer transactions over Parquet, reducing the need for ORC-on-Hive.
package com.bank.formats;
public class WhenToUseEach {
public static void main(String[] args) {
System.out.println("National Bank two-stage pipeline:");
System.out.println(" 1. Kafka -> Avro (streaming, evolving)");
System.out.println(" 2. ETL -> Parquet (curated analytics)");
System.out.println();
System.out.println("Hive regulatory table needing corrections:");
System.out.println(" -> ORC with Hive ACID");
System.out.println();
System.out.println("Modern alternative to ORC-on-Hive:");
System.out.println(" -> Delta Lake or Iceberg on Parquet");
}
}
National Bank two-stage pipeline:
1. Kafka -> Avro (streaming, evolving)
2. ETL -> Parquet (curated analytics)
Hive regulatory table needing corrections:
-> ORC with Hive ACID
Modern alternative to ORC-on-Hive:
-> Delta Lake or Iceberg on Parquet
What are common interview follow-up questions?
- How does Avro schema evolution work differently from Parquet?
- What is the small-files problem and how do you solve it?
- How do Delta Lake and Iceberg provide ACID on Parquet without ORC?
- When would you choose ORC over Parquet today?
- What is the role of a Schema Registry with Avro in Kafka?
Visual Overview
Avro vs Parquet vs ORC ────────────────────── Avro (Row-Based) Parquet (Columnar) ORC (Columnar + ACID) ┌───────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Header: schema│ │ Footer: schema │ │ Tail: schema │ │ [row1 fields] │ │ col1: [v1,v2,...]│ │ col1: [v1,v2,...]│ │ [row2 fields] │ │ col2: [v1,v2,...]│ │ col2: [v1,v2,...]│ │ [row3 fields] │ │ + min/max stats │ │ + indexes + bloom│ │ │ │ + compression │ │ + Hive ACID │ └───────────────┘ └──────────────────┘ └──────────────────┘ Write-friendly Read-friendly Read-friendly + ACID
Two-Stage Pipeline Pattern
──────────────────────────
Kafka ──▶ Avro (write-friendly) ──▶ ETL ──▶ Parquet (read-friendly)
(fast, evolving) (column pruning, pushdown)
Key Takeaway
Avro, Parquet, and ORC differ most fundamentally in physical layout. Avro is row-based, storing each record's fields together with the schema in the header, making it efficient for write-heavy streaming ingestion and giving first-class schema evolution. Parquet is columnar, storing each column's values together, enabling column pruning, predicate pushdown, and strong compression: the default for read-heavy analytics. ORC is also columnar with richer indexes and first-class Hive ACID for row-level updates.
The choice follows the workload. For the write and ingestion path, Avro is the natural fit. For the analytical read path, Parquet is the default across the Spark and cloud ecosystem. ORC is chosen inside the Hive ecosystem when ACID updates are needed. The reason you do not analyze on Avro is that columnar scans read a tiny fraction of bytes; the reason you do not stream into Parquet is that columnar formats want large batched writes.
The common production pattern is a two-stage pipeline: land raw data as Avro for cheap, evolvable writes, then transform and store the curated copy as Parquet for fast reads. Modern lakehouses like Delta and Iceberg layer ACID over Parquet, reducing the need for ORC-on-Hive.
In a banking context at the National Bank of India, high-volume events landed as Avro for fast forward-compatible writes, the curated layer was Parquet for column pruning and pushdown, and a Hive regulatory table used ORC for row-level corrections.
Remember one sentence: Avro to write and evolve, Parquet to read and analyze, ORC when you need Hive ACID, and a two-stage pipeline connects them.
---