What is Parquet | Data Engineering

Q: How is Parquet different from CSV?

package com.bank.formats; public class CsvVsParquet { public static void main(String[] args) { System.out.println("CSV: 100column table, SUM(amount)"); System.out.println(" Reads ALL 100 columns. No skip."); System.out.println(" Size: 10 GB uncompressed."); System.out.println(); System.out.println("

Q: What are the limitations of Parquet?

Parquet is designed for writeonce, readmany analytical use, not for frequent singlerow updates. It pairs with lake table formats like Delta Lake and Iceberg that add ACID transactions and updates on top of Parquet files. It is also poor for retrieving one whole row with all columns, where a row stor

Definition

Parquet stores data by column so analytics reads only the columns it needs, skips irrelevant data, and compresses far better than row formats.

Apache Parquet – an open-source columnar storage file format optimized for analytical workloads.
Columnar layout – storing all values of one column together, instead of all columns of one row together.
Column pruning – reading only the columns a query needs, skipping the rest entirely.
Predicate pushdown – using per-row-group min/max statistics to skip blocks that cannot match a filter.
Row group – a horizontal partition of the data; each row group contains column chunks for every column.
Self-describing – the file footer holds the full schema and per-column statistics.

Imagine the National Bank of India keeps a giant ledger with a hundred columns per transaction: account id, amount, timestamp, branch, device, and ninety-six more. A row-based file is like a book where each page holds one complete transaction with all hundred fields, so to total just the amount column you must flip through every page reading all hundred fields. Parquet is like re-filing the ledger so all the amounts sit together in one slim folder, all the branches in another. Now to total amounts you grab just the amount folder and ignore the other ninety-nine: far less reading, and since all amounts look alike they squash down tiny when zipped.

Understanding the Concepts

What is columnar layout?

The fundamental idea is storing data by column instead of by row. A row-oriented format like CSV stores all fields of row 1, then all fields of row 2. Parquet flips this: it stores all values of column A together, then all of column B. The immediate payoff for analytics is column pruning.

JAVA

package com.bank.formats;

public class ColumnarVsRow {
    public static void main(String[] args) {
        System.out.println("Row-oriented (CSV):");
        System.out.println("  [T1|SAV2001|15000|2024-10-29|Mumbai]");
        System.out.println("  [T2|CUR3002|25000|2024-10-29|Delhi]");
        System.out.println("  -> SUM(amount): must read ALL columns");
        System.out.println();
        System.out.println("Column-oriented (Parquet):");
        System.out.println("  txn_id: [T1, T2, T3, ...]");
        System.out.println("  amount: [15000, 25000, 30000, ...]");
        System.out.println("  date:   [Oct-29, Oct-29, ...]");
        System.out.println("  -> SUM(amount): reads ONLY amount column");
    }
}

TEXT

Row-oriented (CSV):
  [T1|SAV2001|15000|2024-10-29|Mumbai]
  [T2|CUR3002|25000|2024-10-29|Delhi]
  -> SUM(amount): must read ALL columns

Column-oriented (Parquet):
  txn_id: [T1, T2, T3, ...]
  amount: [15000, 25000, 30000, ...]
  date:   [Oct-29, Oct-29, ...]
  -> SUM(amount): reads ONLY amount column

How does column pruning work?

Column pruning means reading only the columns a query needs. On a 100-column transaction table, if the query only selects amount and date, Parquet reads just those two columns' bytes and never touches the other 98, cutting I/O by orders of magnitude.

PYTHON

# Column pruning: Spark reads only 2 of 100 columns
df = spark.read.parquet(
    "s3://national-bank-lake/curated/transactions/")
result = df.select("amount", "txn_date") \
    .groupBy("txn_date").sum("amount")

result.explain(True)
# FileScan parquet [amount, txn_date]
# -> only 2 columns read, 98 skipped

TEXT

== Physical Plan ==
HashAggregate(keys=[txn_date], functions=[sum(amount)])
+- FileScan parquet [amount, txn_date]  <-- only 2 of 100 columns

How does predicate pushdown work?

Parquet organizes data into row groups, and within each row group each column chunk stores min and max values. If a query filters amount > 100000 and a row group's max amount is 50000, the engine skips that entire row group without decoding it.

JAVA

package com.bank.formats;

public class PredicatePushdown {
    public static void main(String[] args) {
        System.out.println("Filter: amount > 100000");
        System.out.println();
        System.out.println("Row group 0: min=500, max=50000");
        System.out.println("  -> max < 100000, SKIP (no decompress)");
        System.out.println();
        System.out.println("Row group 1: min=45000, max=200000");
        System.out.println("  -> max >= 100000, READ");
        System.out.println();
        System.out.println("Row group 2: min=100, max=30000");
        System.out.println("  -> max < 100000, SKIP");
        System.out.println();
        System.out.println("Result: read 1 of 3 row groups.");
    }
}

TEXT

Filter: amount > 100000

Row group 0: min=500, max=50000
  -> max < 100000, SKIP (no decompress)

Row group 1: min=45000, max=200000
  -> max >= 100000, READ

Row group 2: min=100, max=30000
  -> max < 100000, SKIP

Result: read 1 of 3 row groups.

How does compression work in Parquet?

Because a column holds values of the same type and often with repetition, Parquet applies encodings like dictionary encoding and run-length encoding, then a codec like Snappy or Zstandard. Similar values compress dramatically better than mixed-type rows.

JAVA

package com.bank.formats;

public class ParquetCompression {
    public static void main(String[] args) {
        System.out.println("Encoding examples:");
        System.out.println("  branch column: [Mumbai, Mumbai, Delhi,"
            + " Delhi, Mumbai]");
        System.out.println("  Dictionary: {0=Mumbai, 1=Delhi}");
        System.out.println("  Encoded: [0, 0, 1, 1, 0]");
        System.out.println("  -> 5 ints vs 5 strings");
        System.out.println();
        System.out.println("  date column: [2024-10-29 x 1000]");
        System.out.println("  Run-length: (2024-10-29, 1000)");
        System.out.println("  -> 1 value + count vs 1000 values");
        System.out.println();
        System.out.println("Typical compression: 5-10x smaller.");
    }
}

TEXT

Encoding examples:
  branch column: [Mumbai, Mumbai, Delhi, Delhi, Mumbai]
  Dictionary: {0=Mumbai, 1=Delhi}
  Encoded: [0, 0, 1, 1, 0]
  -> 5 ints vs 5 strings

  date column: [2024-10-29 x 1000]
  Run-length: (2024-10-29, 1000)
  -> 1 value + count vs 1000 values

Typical compression: 5-10x smaller.

What is the Parquet file structure?

A Parquet file has a header (magic bytes), one or more row groups, and a footer. The footer holds the full schema and per-column statistics. Engines read the footer first to plan the scan.

DIAGRAM

Parquet File Structure
┌────────────────────────────────┐
│ Header: PAR1 (magic bytes)     │
├────────────────────────────────┤
│ Row Group 0                    │
│   Column Chunk: txn_id         │
│   Column Chunk: amount         │
│   Column Chunk: branch         │
│   (each chunk has pages +      │
│    min/max statistics)         │
├────────────────────────────────┤
│ Row Group 1                    │
│   Column Chunk: txn_id         │
│   Column Chunk: amount         │
│   Column Chunk: branch         │
├────────────────────────────────┤
│ Footer                         │
│   Schema (column names, types) │
│   Row group locations          │
│   Per-column min/max stats     │
├────────────────────────────────┤
│ Footer Length (4 bytes)        │
│ Footer: PAR1 (magic bytes)     │
└────────────────────────────────┘

JAVA

package com.bank.formats;

public class ParquetStructure {
    public static void main(String[] args) {
        System.out.println("1. Engine reads footer first.");
        System.out.println("2. Footer has schema + statistics.");
        System.out.println("3. Engine plans: which columns,"
            + " which row groups.");
        System.out.println("4. Reads only needed data.");
    }
}

TEXT

1. Engine reads footer first.
2. Footer has schema + statistics.
3. Engine plans: which columns, which row groups.
4. Reads only needed data.

How is Parquet different from CSV?

| Aspect | CSV | Parquet |
|---|---|---|
| Layout | Row-oriented | Column-oriented |
| Schema | No (or header row) | Embedded in footer |
| Types | All strings | Typed (int, decimal, etc.) |
| Compression | Weak | Strong (dict, RLE + codec) |
| Column pruning | No (read all) | Yes (read only needed) |
| Predicate pushdown | No | Yes (min/max stats) |
| Human-readable | Yes | No (binary) |
| Best for | Small data, debugging | Analytics at scale |

JAVA

package com.bank.formats;

public class CsvVsParquet {
    public static void main(String[] args) {
        System.out.println("CSV: 100-column table, SUM(amount)");
        System.out.println("  Reads ALL 100 columns. No skip.");
        System.out.println("  Size: 10 GB uncompressed.");
        System.out.println();
        System.out.println("Parquet: same table, SUM(amount)");
        System.out.println("  Reads ONLY amount column. 99 skipped.");
        System.out.println("  Size: 1.5 GB compressed.");
    }
}

TEXT

CSV: 100-column table, SUM(amount)
  Reads ALL 100 columns. No skip.
  Size: 10 GB uncompressed.

Parquet: same table, SUM(amount)
  Reads ONLY amount column. 99 skipped.
  Size: 1.5 GB compressed.

What are the limitations of Parquet?

Parquet is designed for write-once, read-many analytical use, not for frequent single-row updates. It pairs with lake table formats like Delta Lake and Iceberg that add ACID transactions and updates on top of Parquet files. It is also poor for retrieving one whole row with all columns, where a row store still wins.

JAVA

package com.bank.formats;

public class ParquetLimitations {
    public static void main(String[] args) {
        System.out.println("Parquet strengths:");
        System.out.println("  Wide scans, aggregations, analytics");
        System.out.println();
        System.out.println("Parquet weaknesses:");
        System.out.println("  No in-place UPDATE/DELETE");
        System.out.println("  Poor for single-row full-column fetch");
        System.out.println("  Not human-readable (binary)");
        System.out.println();
        System.out.println("Solution: Delta Lake / Iceberg on top");
        System.out.println("  Adds ACID, MERGE, time travel.");
    }
}

TEXT

Parquet strengths:
  Wide scans, aggregations, analytics

Parquet weaknesses:
  No in-place UPDATE/DELETE
  Poor for single-row full-column fetch
  Not human-readable (binary)

Solution: Delta Lake / Iceberg on top
  Adds ACID, MERGE, time travel.

What are common interview follow-up questions?

How does sorting the data improve Parquet predicate pushdown?
What is the difference between Parquet and ORC?
How does Delta Lake add ACID on top of Parquet?
What is the difference between Snappy and Zstandard compression?
How do you choose the row group size?

Visual Overview

DIAGRAM

  Row-Store vs Column-Store (Parquet)
  ───────────────────────────────────

  Row-Store (CSV)                  Column-Store (Parquet)
  ┌──────────────────┐             ┌──────────────────────┐
  │ T1|15000|Oct|Mum │             │ txn_id: T1,T2,T3,... │
  │ T2|25000|Oct|Del │             │ amount: 15k,25k,...   │
  │ T3|30000|Oct|Blr │             │ date:   Oct,Oct,...   │
  │ (all cols per row)│            │ branch: Mum,Del,...   │
  └──────────────────┘             └──────────────────────┘

  SELECT SUM(amount):              SELECT SUM(amount):
  Read ALL columns :(              Read ONLY amount column :)

DIAGRAM

  Parquet Scan Optimization
  ─────────────────────────
  Query: SELECT SUM(amount) WHERE date = '2024-10-29'

  Step 1: Column pruning
    -> Read only 'amount' and 'date' columns

  Step 2: Predicate pushdown
    -> Row group 0: date max = Sep  -> SKIP
    -> Row group 1: date max = Oct  -> READ
    -> Row group 2: date max = Nov  -> SKIP

  Step 3: Decompress + aggregate
    -> Only the surviving rows

Key Takeaway

Apache Parquet is an open-source columnar storage file format built for analytics. Instead of storing data row by row, it stores values column by column, and that one change unlocks three big wins. Column pruning lets a query read just the columns it needs, cutting I/O by orders of magnitude on wide tables. Predicate pushdown uses per-row-group min/max statistics to skip entire blocks that cannot match a filter. And compression is far better because same-type column values encode efficiently via dictionary and run-length encoding plus a codec like Snappy.

Parquet is self-describing, with a footer holding the schema and statistics, which is why engines read the footer first to plan the scan. It integrates natively with Spark, Trino, Hive, and cloud warehouses. The trade-off is that Parquet is designed for write-once, read-many analytical use, not frequent single-row updates, which is why it pairs with lake table formats like Delta Lake and Iceberg for ACID and updates.

In a banking context at the National Bank of India, transaction history lived in partitioned Parquet in the data lake. A report needing total amounts read only the amount column and ignored dozens of other columns, and a report scoped to one date skipped every row group outside that date using min/max statistics.

The result was that analytical queries like per-branch quarterly totals were far cheaper in storage and far faster to run than they would have been in a row format.

Remember one sentence: Parquet stores data by column so analytics reads only what it needs, skips irrelevant blocks, and compresses hard.

---