Definition
Effective bucketing has specific requirements and important caveats.
- High cardinality – The column must have many distinct values for hashing to distribute evenly. Account ID is good; branch code is not.
- Aligned with join/group keys – Bucketing only pays off on the column you actually join or group on.
- Sensible bucket count – Sized to cluster parallelism and data volume; fixed at write time.
- Catalog requirement – Metadata lives only in the metastore via
saveAsTable. Raw path writes lose the benefit. - Identical sides – Both tables in a bucket-aware join must share the same bucket count and column.
- No Delta support – Delta tables use OPTIMIZE, ZORDER, and liquid clustering instead of Hive-style bucketing.
At the National Bank of India, bucketing is useful only when you pick the right label rule and register it with the records office. You bucket by account number: the thing you constantly match and total, not by branch, which has too few values to spread evenly. You tell the records office exactly how many drawers you are using and by what rule, because if the office does not have that on file, the next clerk will re-sort everything anyway. And if you switched to a new filing cabinet brand (Delta), its drawers do not follow the old rule at all.
Understanding the Concepts
What is bucketing?
Bucketing spreads rows across a fixed number of files by hashing a column. When two tables are bucketed on the same column with the same count, a join between them can skip the shuffle entirely because matching rows are already co-located.
# Write a bucketed table
txns = spark.read.parquet("/data/bank/raw/transactions")
txns.write \
.bucketBy(128, "account_id") \
.sortBy("account_id") \
.saveAsTable("banking_prod.transactions_bucketed")
accounts = spark.read.parquet("/data/bank/raw/accounts")
accounts.write \
.bucketBy(128, "account_id") \
.sortBy("account_id") \
.saveAsTable("banking_prod.accounts_bucketed")
Why must the column be high cardinality?
Bucketing hashes the column value to determine which bucket (file) the row goes into. Only a column with many distinct values distributes evenly across buckets. Low cardinality concentrates rows in a few buckets.
# Good: account_id has millions of distinct values → even distribution
txns.select("account_id").distinct().count() # 2,000,000
# Bad: branch_code has only 50 values → most buckets empty
txns.select("branch_code").distinct().count() # 50
# With 128 buckets and 50 distinct branch codes,
# 78 buckets would be empty and 50 would be unevenly loaded
How do you choose the bucket count?
The bucket count determines parallelism and file sizes. Too few gives huge files and low parallelism; too many gives tiny files and overhead.
# Rule of thumb: bucket count near the number of cores
# or sized so each bucket is 100-500 MB
total_data_size_gb = 100 # 100 GB of transactions
target_bucket_size_mb = 256
bucket_count = int((total_data_size_gb * 1024) / target_bucket_size_mb)
print(f"Recommended bucket count: {bucket_count}") # ~400
# The count is fixed at write time
# If data grows significantly, you must rewrite with a larger count
Why must you use saveAsTable?
Bucketing metadata (column, count, sort order) lives only in the metastore, not in the files. If you write by raw path, Spark sees ordinary files with no bucket information.
# CORRECT: register through saveAsTable
txns.write.bucketBy(128, "account_id").sortBy("account_id") \
.saveAsTable("banking_prod.transactions_bucketed")
# Read through catalog: optimizer knows the bucketing layout
df = spark.table("banking_prod.transactions_bucketed")
# WRONG: write by raw path -- metadata lost!
txns.write.bucketBy(128, "account_id").sortBy("account_id") \
.parquet("/data/bank/transactions_bucketed")
# Reading by path: Spark sees ordinary Parquet, shuffles anyway
df = spark.read.parquet("/data/bank/transactions_bucketed")
Why must both join sides be identically bucketed?
For a shuffle-free join, both tables must have the same bucket count and column. If they differ, Spark must re-shuffle one or both sides.
# Both sides: 128 buckets on account_id → shuffle-free join
result = spark.table("banking_prod.transactions_bucketed") \
.join(spark.table("banking_prod.accounts_bucketed"), "account_id")
result.explain()
# Physical plan shows NO Exchange (shuffle) node
Scenario 1: Same bucket count and column txns: 128 buckets on account_id accounts: 128 buckets on account_id → Shuffle-free join (bucket-to-bucket) Scenario 2: Different bucket counts txns: 128 buckets on account_id accounts: 64 buckets on account_id → Spark re-shuffles one side (bucketing benefit lost) Scenario 3: Different columns txns: 128 buckets on account_id accounts: 128 buckets on customer_id → Spark re-shuffles both sides (no co-location)
Why does Delta not support bucketing?
Delta Lake tables do not support classic Hive-style bucketing. Delta uses its own data-skipping and clustering mechanisms instead.
-- On Delta tables, use OPTIMIZE and ZORDER instead of bucketing
OPTIMIZE banking_prod.transactions
ZORDER BY (account_id);
-- Or use liquid clustering (Databricks)
ALTER TABLE banking_prod.transactions
CLUSTER BY (account_id);
Hive Bucketing: Fixed number of files, hash-based, catalog-dependent
Delta ZORDER: Dynamic file layout, multi-column co-location, auto-maintained
Liquid Clustering: Incremental, auto-tuning, no manual OPTIMIZE needed
How do you verify bucketing is working?
Check the physical plan for the absence of Exchange (shuffle) nodes.
# Verify shuffle-free join
result = spark.table("banking_prod.transactions_bucketed") \
.join(spark.table("banking_prod.accounts_bucketed"), "account_id")
result.explain(True)
# Look for: NO Exchange node before the join
# If you see Exchange, bucketing is not being used
What should a senior engineer know?
- Bucketing is a long-lived physical decision: rebucketing a large table is a full rewrite.
- Only commit to bucketing for stable, frequently joined keys.
- On Delta lakehouses, use ZORDER or liquid clustering, not bucketing.
- Bucketing helps joins and aggregations on the bucketed column but does nothing for other columns.
- New Spark versions relax the requirement that both sides must also be sorted, but verify in the plan.
# Anti-pattern: bucketing a column nobody joins on
txns.write.bucketBy(128, "txn_date").saveAsTable("txns_by_date")
# Wastes write time if you only join on account_id
# Correct: bucket on the key you actually join on
txns.write.bucketBy(128, "account_id").saveAsTable("txns_by_account")
Visual Overview
Bucketed Join (shuffle-free):
Transactions (128 buckets) Accounts (128 buckets)
┌────────┐ ┌────────┐
│Bucket 0│ ◄──── account_id ───►│Bucket 0│
│Bucket 1│ ◄──── account_id ───►│Bucket 1│
│Bucket 2│ ◄──── account_id ───►│Bucket 2│
│ ... │ │ ... │
│Bucket127│◄──── account_id ───►│Bucket127│
└────────┘ └────────┘
│ │
└──────── Direct Join ──────────┘
(NO shuffle needed)
Unbucketed Join (with shuffle):
┌──────────┐ ┌──────────┐
│ Txns │ │ Accounts │
│ (random) │ │ (random) │
└────┬─────┘ └────┬─────┘
│ │
▼ ▼
┌──────────────────────────┐
│ SHUFFLE (expensive) │
│ Repartition by account_id │
└──────────────────────────┘
│
▼
Join (after shuffle)
Key Takeaway
Effective bucketing has two requirements: the column must be high cardinality for even distribution across buckets, and it must be the column you actually join or group on. Account ID with millions of distinct values is a good choice; branch code with fifty values is not. You also pick a bucket count sized to your cluster parallelism and data volume, and you remember it is fixed at write time.
The central caveat is the catalog. Bucketing metadata lives only in the metastore, so you must write with saveAsTable and read through the catalog table. Writing by raw path loses the metadata and Spark shuffles anyway. Both sides of a bucket-aware join must share the identical bucket count and column, or Spark re-shuffles. And on Delta lakehouses, classic Hive-style bucketing is not supported: use OPTIMIZE with ZORDER or liquid clustering instead.
At the National Bank of India we bucketed the transactions and accounts tables by account_id into 128 buckets via saveAsTable, making the daily join shuffle-free. We corrected a colleague who bucketed by branch (too low cardinality) and another who wrote by raw path (metadata lost). When we later moved to Delta, we switched from bucketing to ZORDER clustering.
Remember one sentence: bucket high-cardinality join keys with identical counts on both sides via saveAsTable, and on Delta use ZORDER or liquid clustering instead.
---