Definition
- Min/max statistics – per-column minimum and maximum values stored at row-group and page level, enabling range-based block skipping.
- Dictionary filtering – checking the dictionary page of a dictionary-encoded column to see if a filter value exists at all in a block.
- Bloom filter – a compact probabilistic bitset that answers "definitely not present" or "possibly present" for high-cardinality equality checks.
- Row group – a horizontal partition of the Parquet file containing column chunks for all columns.
- Column chunk – one column's data within one row group, containing pages.
- Sort order – the arrangement of data that determines whether min/max ranges are tight (useful) or wide (useless).
The National Bank of India's archive has three ways to avoid opening boxes. Each box has a label saying the smallest and largest account balance inside (min/max): if you want balances above a lakh and the box tops out at fifty thousand, skip it. Each box also has an index card listing the distinct transaction types inside (dictionary): if "wire transfer" is not on the card, skip the box. And each box has a quick fingerprint scanner (bloom filter) that says "this account id is definitely not in here." If the boxes were filled in sorted order by balance, the min/max labels are tight and precise; if filled randomly, every box spans the whole range and the labels become useless.
Understanding the Concepts
How do min/max statistics work?
A Parquet file is split into row groups, each row group into column chunks, and each column chunk into pages. At both levels, Parquet stores the minimum and maximum value of each column. When Spark pushes a predicate, the reader compares it to each block's range and skips any block that cannot match.
package com.bank.formats;
public class MinMaxStatistics {
public static void main(String[] args) {
System.out.println("Filter: amount > 100000");
System.out.println();
System.out.println("Row Group 0: min=500, max=50000");
System.out.println(" 50000 < 100000 -> SKIP (no decompress)");
System.out.println();
System.out.println("Row Group 1: min=45000, max=200000");
System.out.println(" 200000 >= 100000 -> READ");
System.out.println();
System.out.println("Row Group 2: min=100, max=30000");
System.out.println(" 30000 < 100000 -> SKIP");
System.out.println();
System.out.println("Result: read 1 of 3 row groups.");
System.out.println("Saved: 67% I/O skipped.");
}
}
Filter: amount > 100000
Row Group 0: min=500, max=50000
50000 < 100000 -> SKIP (no decompress)
Row Group 1: min=45000, max=200000
200000 >= 100000 -> READ
Row Group 2: min=100, max=30000
30000 < 100000 -> SKIP
Result: read 1 of 3 row groups.
Saved: 67% I/O skipped.
How does dictionary filtering work?
Parquet often dictionary-encodes a column, storing distinct values once in a dictionary page. For an equality filter like txn_type = 'wire', the reader checks the dictionary page: if 'wire' is not in the dictionary, no row can match, and the block is skipped.
package com.bank.formats;
public class DictionaryFiltering {
public static void main(String[] args) {
System.out.println("Block's dictionary page for txn_type:");
System.out.println(" {0: 'card', 1: 'upi', 2: 'neft'}");
System.out.println();
System.out.println("Filter: txn_type = 'wire'");
System.out.println(" 'wire' NOT in dictionary -> SKIP block");
System.out.println();
System.out.println("Filter: txn_type = 'upi'");
System.out.println(" 'upi' IN dictionary -> READ block");
System.out.println();
System.out.println("Effective for low-to-medium cardinality.");
}
}
Block's dictionary page for txn_type:
{0: 'card', 1: 'upi', 2: 'neft'}
Filter: txn_type = 'wire'
'wire' NOT in dictionary -> SKIP block
Filter: txn_type = 'upi'
'upi' IN dictionary -> READ block
Effective for low-to-medium cardinality.
How do bloom filters work?
A bloom filter is a compact bitset where each value is hashed into several bit positions. To test: hash the value and check those bits. If any bit is zero, the value is definitely not present and the block is skipped. If all bits are set, the value is possibly present and the block is read. No false negatives, only false positives.
package com.bank.formats;
public class BloomFilterExplained {
public static void main(String[] args) {
System.out.println("Bloom filter for txn_id column:");
System.out.println(" Bitset: [0,1,1,0,1,0,0,1,...]");
System.out.println();
System.out.println("Test txn_id = 'T99999':");
System.out.println(" hash1('T99999') -> bit 3 = 0");
System.out.println(" Definitely NOT present -> SKIP block");
System.out.println();
System.out.println("Test txn_id = 'T10045':");
System.out.println(" hash1 -> bit 1 = 1");
System.out.println(" hash2 -> bit 4 = 1");
System.out.println(" hash3 -> bit 7 = 1");
System.out.println(" Possibly present -> READ block");
System.out.println();
System.out.println("Key: no false negatives, safe for skipping.");
}
}
Bloom filter for txn_id column:
Bitset: [0,1,1,0,1,0,0,1,...]
Test txn_id = 'T99999':
hash1('T99999') -> bit 3 = 0
Definitely NOT present -> SKIP block
Test txn_id = 'T10045':
hash1 -> bit 1 = 1
hash2 -> bit 4 = 1
hash3 -> bit 7 = 1
Possibly present -> READ block
Key: no false negatives, safe for skipping.
Why is sort order critical for min/max?
Min/max statistics only help if the per-block ranges are tight and non-overlapping. If data is sorted by the filter column, each block holds a narrow slice. If unsorted, every block spans the whole domain and no block is ever skipped.
package com.bank.formats;
public class SortOrderMatters {
public static void main(String[] args) {
System.out.println("SORTED by amount:");
System.out.println(" Block 0: [0, 10000]");
System.out.println(" Block 1: [10001, 20000]");
System.out.println(" Block 2: [20001, 50000]");
System.out.println(" Filter: amount > 30000");
System.out.println(" -> Blocks 0,1 SKIPPED. Block 2 READ.");
System.out.println();
System.out.println("UNSORTED:");
System.out.println(" Block 0: [50, 49000]");
System.out.println(" Block 1: [100, 48000]");
System.out.println(" Block 2: [75, 50000]");
System.out.println(" Filter: amount > 30000");
System.out.println(" -> ALL blocks read. Stats useless.");
}
}
SORTED by amount:
Block 0: [0, 10000]
Block 1: [10001, 20000]
Block 2: [20001, 50000]
Filter: amount > 30000
-> Blocks 0,1 SKIPPED. Block 2 READ.
UNSORTED:
Block 0: [50, 49000]
Block 1: [100, 48000]
Block 2: [75, 50000]
Filter: amount > 30000
-> ALL blocks read. Stats useless.
How do the three mechanisms compare?
| Mechanism | Type | Predicate | Cardinality | Default | Cost |
|---|---|---|---|---|---|
| Min/max | Exact range | Range (>, <, =) | Any | On | None (in footer) |
| Dictionary | Exact membership | Equality (=) | Low-medium | On (if encoded) | None |
| Bloom filter | Probabilistic | Equality (=) | High | Off (opt-in) | Space at write |
package com.bank.formats;
public class ThreeWayComparison {
public static void main(String[] args) {
System.out.println("When each mechanism shines:");
System.out.println(" Min/max: range filters on sorted columns");
System.out.println(" Dictionary: equality on low-card columns");
System.out.println(" Bloom: equality on high-card columns");
}
}
When each mechanism shines:
Min/max: range filters on sorted columns
Dictionary: equality on low-card columns
Bloom: equality on high-card columns
How do you enable bloom filters?
Bloom filters must be explicitly enabled per column at write time because they cost space. You enable them only for high-cardinality columns you filter by equality.
# Enable bloom filter on txn_id at write time
df.write.format("parquet") \
.option("parquet.bloom.filter.enabled#txn_id", "true") \
.option("parquet.bloom.filter.expected.ndv#txn_id", "1000000") \
.save("s3://bank-lake/curated/transactions/")
package com.bank.formats;
public class EnableBloomFilter {
public static void main(String[] args) {
System.out.println("Bloom filter config:");
System.out.println(" Enabled per column at write time.");
System.out.println(" Costs space (extra bytes per block).");
System.out.println(" Enable only for high-cardinality"
+ " equality-filtered columns.");
System.out.println(" Example: txn_id, account_id");
System.out.println(" Not needed for: branch (use dictionary)");
}
}
Bloom filter config:
Enabled per column at write time.
Costs space (extra bytes per block).
Enable only for high-cardinality equality-filtered columns.
Example: txn_id, account_id
Not needed for: branch (use dictionary)
How do you verify pushdown in Spark?
Check the physical plan for PushedFilters to verify that predicates are being pushed to the Parquet reader.
df = spark.read.parquet("s3://bank-lake/curated/transactions/")
result = df.filter("amount > 100000").select("txn_id", "amount")
result.explain(True)
== Physical Plan ==
*(1) Filter (amount > 100000)
+- FileScan parquet [txn_id, amount]
PushedFilters: [GreaterThan(amount, 100000)]
ReadSchema: struct<txn_id:string, amount:decimal(15,2)>
What are common interview follow-up questions?
- What is Z-ordering and how does it help multi-column predicate pushdown?
- How do you choose between Snappy and Zstandard compression?
- What is the optimal row group size for pushdown effectiveness?
- How does Delta Lake's data skipping build on Parquet statistics?
- What is the difference between page-level and row-group-level statistics?
Visual Overview
Three Pushdown Mechanisms
─────────────────────────
1. Min/Max (range) 2. Dictionary (equality) 3. Bloom (equality, high-card)
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Block stats: │ │ Dict page: │ │ Bloom bitset:│
│ min=500 │ │ {card,upi, │ │ [0,1,1,0,1,0]│
│ max=50000 │ │ neft} │ │ │
│ │ │ │ │ hash(T99999) │
│ Filter: │ │ Filter: │ │ -> bit 3 = 0 │
│ amt > 100000 │ │ type='wire' │ │ -> NOT present│
│ 50000<100000 │ │ wire not in │ │ -> SKIP │
│ -> SKIP │ │ dict -> SKIP │ │ │
└──────────────┘ └──────────────┘ └──────────────┘
Sorted vs Unsorted ────────────────── Sorted: [0-10k] [10k-20k] [20k-50k] <- tight ranges, blocks skip Unsorted: [50-49k] [100-48k] [75-50k] <- wide overlap, nothing skips
Key Takeaway
Parquet implements predicate pushdown through three layered data-skipping mechanisms. Min/max statistics are stored at both row-group and page level, so the reader compares filters to those ranges and skips any block that cannot match. Dictionary filtering checks the dictionary page for equality predicates: if the filter value is not in the dictionary, no row can match. Bloom filters are compact probabilistic bitsets for high-cardinality columns, giving "definitely not present" which is safe for skipping.
Sort order is the key to min/max effectiveness. If data is sorted by the filter column, each block holds a narrow slice of values, so filters cleanly isolate blocks. If unsorted, every block spans the whole range and nothing is ever skipped: the statistics exist but are worthless. That is why data is sorted or Z-ordered on frequently filtered columns before writing.
Min/max and dictionary filtering are on by default, while bloom filters must be explicitly enabled per column at write time because they cost space. And all of this depends on the predicate being pushable, which you verify through the physical plan's PushedFilters.
In a banking context at the National Bank of India, transaction tables were sorted by amount and date so those filters could skip most blocks by min/max, dictionary filtering handled low-cardinality columns like transaction type, and bloom filters were enabled specifically on the high-cardinality transaction id for point lookups.
Remember one sentence: Parquet skips blocks using min/max ranges, dictionary membership, and bloom filters, but sorting the data is what makes min/max actually work.
---