Definition
- Snappy – fast compression and decompression, splittable, default for Parquet output.
- ZSTD – better compression ratio than Snappy with modern speed, increasingly the recommended default for cold data.
- Gzip – good ratio but slow, and a single Gzip text file is not splittable, so one task reads the whole file.
- LZ4 – extremely fast with a modest ratio, good for hot shuffle data; default internal codec in modern Spark.
- Bzip2 – best ratio but slowest; splittable for text but rarely used except for archival.
- Two layers – output format compression (per writer) and internal engine compression (shuffle, broadcast, cache, checkpoint, event logs) are configured separately.
Think of the National Bank of India shipping boxes of documents between branches. A codec is how tightly you pack each box. Snappy packs quickly and the boxes can be opened by many workers at once, so it is the everyday default. Bzip2 packs the tightest so you pay the least for shipping, but packing and unpacking is painfully slow. Gzip packs well but each box must be opened by a single worker from the start because it cannot be split. ZSTD is the modern choice that packs tight and still fast.
Understanding the Concepts
What compression codecs does Spark support?
Spark supports several codecs that trade off speed against compression ratio and splittability. The right choice depends on the use case: storage cost, read speed, or write speed.
| Codec | Speed | Ratio | Splittable (text) | Typical Use |
|---|---|---|---|---|
| Snappy | Fast | Moderate | Yes | Default for Parquet, everyday use |
| ZSTD | Fast-Moderate | Good | Yes | Cold data, long-term storage |
| Gzip | Slow | Good | No | Legacy, avoid for large files |
| LZ4 | Very Fast | Low-Moderate | No | Internal shuffle, hot paths |
| Bzip2 | Very Slow | Best | Yes | Archival only |
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CompressionDemo").getOrCreate()
txns = spark.read.parquet("/data/transactions")
How do you set output format compression?
Output compression is set per writer using the compression option. This controls how the data files themselves are compressed on disk.
# Write with Snappy (default for Parquet)
txns.write.option("compression", "snappy") \
.parquet("/output/transactions_snappy")
# Write with ZSTD for better ratio on cold archival data
txns.write.option("compression", "zstd") \
.parquet("/output/transactions_zstd")
# Write CSV with Gzip (common but not ideal for large files)
txns.write.option("compression", "gzip") \
.csv("/output/transactions_gzip")
# ORC with ZSTD
txns.write.option("compression", "zstd") \
.orc("/output/transactions_orc_zstd")
How do you set internal engine compression?
The internal engine compresses temporary data: shuffle files, broadcast variables, cached RDD blocks, checkpoint data, and event logs. This is configured separately from output format compression.
# Set the internal codec (applies to shuffle, broadcast, cache, etc.)
spark.conf.set("spark.io.compression.codec", "lz4") # Default in modern Spark
# Fine-grained toggles for each layer
spark.conf.set("spark.shuffle.compress", "true") # Compress shuffle files
spark.conf.set("spark.broadcast.compress", "true") # Compress broadcast vars
spark.conf.set("spark.rdd.compress", "true") # Compress cached RDD blocks
spark.conf.set("spark.checkpoint.compress", "true") # Compress checkpoint data
# Switch internal codec to ZSTD for better shuffle compression
spark.conf.set("spark.io.compression.codec", "zstd")
Why does splittability matter?
Splittability determines whether a large file can be read by multiple tasks in parallel. A non-splittable file forces one task to read the entire file from start to finish, destroying parallelism.
# PROBLEM: a single large Gzip CSV file
# Only ONE task can read this file because Gzip is not splittable
large_gzip = spark.read.csv("/data/archive/all_trades.csv.gz")
large_gzip.rdd.getNumPartitions() # Returns 1 -- no parallelism!
# SOLUTION: use Snappy Parquet which splits at row-group level
# Multiple tasks read different row groups in parallel
large_parquet = spark.read.parquet("/data/archive/all_trades.parquet")
large_parquet.rdd.getNumPartitions() # Returns many -- full parallelism!
# Gzip CSV: 1 task reads 10 GB file sequentially
# Time: ~30 minutes
# Snappy Parquet: 80 tasks read 80 row groups in parallel
# Time: ~45 seconds
How do you choose between Snappy and ZSTD?
The choice depends on whether you optimize for read speed or storage cost.
# Compare file sizes for the same data
import os
txns.write.option("compression", "snappy").parquet("/tmp/snappy")
txns.write.option("compression", "zstd").parquet("/tmp/zstd")
txns.write.option("compression", "gzip").parquet("/tmp/gzip")
txns.write.option("compression", "none").parquet("/tmp/none")
# Typical results for transaction data:
# Uncompressed: 1000 MB
# Snappy: 400 MB (2.5x compression, fastest read)
# ZSTD: 300 MB (3.3x compression, competitive read speed)
# Gzip: 280 MB (3.6x compression, slowest read)
| Scenario | Recommended Codec | Reason |
|---|---|---|
| Hot data queried frequently | Snappy | Fastest decompression |
| Cold archival data stored for years | ZSTD (higher level) | Best ratio at acceptable speed |
| Shuffle data between stages | LZ4 | Fastest, network-bound path |
| Legacy system requiring Gzip | Gzip | Compatibility, but avoid for new work |
What about ZSTD compression levels?
ZSTD supports tunable compression levels that trade more CPU for smaller files. Higher levels are useful for archival banking data retained for years.
# ZSTD level 1: fast, moderate compression (default)
txns.write.option("compression", "zstd") \
.option("io.compression.codec.zstd.level", "1") \
.parquet("/output/zstd_level1")
# ZSTD level 9: slower, tighter compression for archival
txns.write.option("compression", "zstd") \
.option("io.compression.codec.zstd.level", "9") \
.parquet("/output/zstd_level9")
How does compression interact with Parquet row groups?
Inside a Parquet file, compression is applied at the column-chunk level within each row group. This means Parquet files are always splittable at the row-group boundary, regardless of the codec.
Parquet File Structure: ┌─────────────────────────────────────────────────┐ │ Row Group 1 │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Column: account_id│ │ Column: amount │ ... │ │ │ (Snappy compressed)│ │(Snappy compressed)│ │ │ └──────────────────┘ └──────────────────┘ │ ├─────────────────────────────────────────────────┤ │ Row Group 2 │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Column: account_id│ │ Column: amount │ ... │ │ │ (Snappy compressed)│ │(Snappy compressed)│ │ │ └──────────────────┘ └──────────────────┘ │ └─────────────────────────────────────────────────┘ Task 1 reads Row Group 1 ← parallel Task 2 reads Row Group 2 ← parallel
What should a senior engineer know?
- Always set compression explicitly in production so it does not change when Spark versions upgrade.
- Snappy or ZSTD Parquet for the lake keeps reads parallel and column-pruned.
- LZ4 for internal paths because shuffle and broadcast are speed-sensitive.
- Gzip CSV is a legacy trap – at the National Bank of India, years of settled trades stored as Gzip CSV were barely parallel on analytical scans; switching to ZSTD Parquet made them smaller and fully splittable.
- Compressing shuffle is almost always a win because shuffle is network- and disk-bound, so the CPU cost of LZ4 compression is repaid many times over by smaller network transfers.
Visual Overview
Two Separate Compression Layers in Spark
Layer 1: Output Format (per writer)
┌──────────────────────────────────────────┐
│ df.write.option("compression", "zstd") │
│ .parquet("/data/lake/transactions") │
│ │
│ Controls: data files on disk │
│ Codecs: snappy, zstd, gzip, none │
└──────────────────────────────────────────┘
Layer 2: Internal Engine (spark.io.compression.codec)
┌──────────────────────────────────────────┐
│ spark.io.compression.codec = lz4 │
│ │
│ Controls: │
│ ├── Shuffle files between stages │
│ ├── Broadcast variables │
│ ├── Cached RDD blocks (serialized) │
│ ├── Checkpoint data │
│ └── Event logs │
└──────────────────────────────────────────┘
Splittability Comparison Gzip CSV (NOT splittable) ┌───────────────────────────────────────┐ │ 10 GB single file │ │ Task 1 reads from start to end │ │ No parallel reading possible │ └───────────────────────────────────────┘ Snappy Parquet (splittable at row groups) ┌──────────┐┌──────────┐┌──────────┐┌──────────┐ │ RG 1 ││ RG 2 ││ RG 3 ││ RG 4 │ │ Task 1 ││ Task 2 ││ Task 3 ││ Task 4 │ └──────────┘└──────────┘└──────────┘└──────────┘ Full parallelism!
Key Takeaway
Spark supports several compression codecs, and the choice is a trade-off between speed, compression ratio, and splittability. Snappy is fast and splittable and is the default for Parquet. ZSTD gives a better ratio at competitive speed and is the modern choice for cold data. Gzip gives a good ratio but is slow and not splittable for text files, which destroys parallelism on large single files. LZ4 is the fastest with a modest ratio, ideal for internal shuffle paths.
The critical detail is that there are two separate compression layers. Output format compression is set per writer and controls how data files are stored on disk. Internal engine compression, controlled by spark.io.compression.codec, covers shuffle files, broadcast variables, cached RDD blocks, checkpoint data, and event logs. These are configured independently.
At the National Bank of India, years of settled trades stored as large Gzip CSV files were barely parallel on analytical scans because each file was read by a single task. Switching to ZSTD Parquet made the files even smaller and fully splittable, so scans used the whole cluster. Separately, enabling shuffle compression with LZ4 on a network-bound pipeline cut the bytes crossing the network and sped up stage boundaries.
The rules for production are: use Snappy or ZSTD Parquet for storage to keep reads parallel, reach for ZSTD at a higher level only when long-term storage cost dominates, keep the internal codec on a fast option like LZ4 for shuffle and broadcast, and always set compression explicitly so it does not change under you.
Remember one sentence: Snappy or ZSTD for splittable storage files, LZ4 for fast internal paths, and never store large analytical data as Gzip CSV because it cannot be parallelized.
---