Back to Apache Spark: Core
Q42Question 42 of 72

Bucketing (`bucketBy`) and how it differs from partitioning

Key Concepts
  • Bucketing hashes a column into a fixed number of files so the same key always lands in the same bucket.
  • Unlike partitioning, bucketing produces a fixed file count regardless of column cardinality, avoiding directory explosion.
  • Two tables bucketed on the same column with the same bucket count can be joined without a shuffle.
  • Bucketing requires saveAsTable because metadata must be registered in the catalog.
  • Bucketing is ideal for high-cardinality join/group keys; partitioning is ideal for low-cardinality filter columns.
  • You can combine partitioning and bucketing: partition by low-cardinality for pruning, bucket by high-cardinality for shuffle-free joins.

Definition

Bucketing hashes a column into a fixed number of files so the same key always lands in the same bucket, giving join and aggregation benefits without the directory explosion of partitioning.
  • Bucketing – divides a dataset into a fixed number of files (buckets) by applying hash(column) % numBuckets, so all rows with the same key always land in the same bucket.
  • Declared with df.write.bucketBy(n, "col").sortBy("col2").saveAsTable(name) – must use saveAsTable to register metadata in the catalog.
  • Key difference from partitioning – partitioning creates a variable number of directories (one per distinct value); bucketing creates a fixed number of files regardless of cardinality.
  • Benefit – two tables with matching bucket count and column can be joined without a shuffle.

At the National Bank of India, partitioning is like labelling drawers by branch – the number of drawers grows with the number of branches. Bucketing is like deciding up front to use exactly thirty-two drawers and dropping each account's records into a drawer chosen by a fixed rule, like the last digits of the account number. No matter how many accounts exist, there are always exactly thirty-two drawers, and the same account always goes to the same drawer.

Understanding the Concepts

What is bucketing in Spark?

Bucketing assigns each row to a bucket using hash(bucketColumn) % numBuckets. Because the hash is deterministic, every row with the same key value lands in the same bucket number. The number of buckets is fixed by you, not by the data. You declare it at write time with bucketBy and must use saveAsTable.

PYTHON
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder \
    .appName("NBI-Bucketing") \
    .config("spark.sql.warehouse.dir", "/bank/warehouse") \
    .enableHiveSupport() \
    .getOrCreate()

txn = spark.createDataFrame([
    ("SAV2001", "Mumbai", 50000.00, "2026-01-15"),
    ("SAV2001", "Mumbai", 75000.00, "2026-01-16"),
    ("SAV2002", "Delhi",  12000.00, "2026-01-15"),
    ("CUR3001", "Chennai", 300000.00, "2026-01-16"),
    ("SAV2002", "Delhi",  8000.00, "2026-01-17")
], ["account_id", "branch", "amount", "txn_date"])

# Bucket by account_id into 4 buckets, sorted by txn_date
txn.write.bucketBy(4, "account_id").sortBy("txn_date") \
    .saveAsTable("bank.transactions_bucketed")
TEXT
Output files:
/bank/warehouse/bank.db/transactions_bucketed/
  part-00000-..._00000.snappy.parquet   (bucket 0)
  part-00001-..._00001.snappy.parquet   (bucket 1)
  part-00002-..._00002.snappy.parquet   (bucket 2)
  part-00003-..._00003.snappy.parquet   (bucket 3)

How does bucketing differ from partitioning?

The fundamental difference is the output shape. Partitioning creates one directory per distinct value, so the count is variable and grows with cardinality. Bucketing creates a fixed number of files set by you, independent of cardinality.

| Feature | Partitioning | Bucketing |
|---|---|---|
| Output shape | Variable directories (one per value) | Fixed number of files |
| Column cardinality | Must be low | Can be high |
| Data skipping | Directory-level pruning | Bucket-level pruning (equality) |
| Shuffle elimination | No | Yes (matching bucket count + column) |
| Storage API | write.partitionBy(col) | write.bucketBy(n, col).saveAsTable |
| Metadata | In directory names | In catalog (Hive metastore) |

PYTHON
# Partitioning: 48 directories (one per branch)
txn.write.partitionBy("branch").parquet("/bank/partitioned/")

# Bucketing: always 4 files (regardless of account count)
txn.write.bucketBy(4, "account_id").sortBy("txn_date") \
    .saveAsTable("bank.txn_bucketed")

How does bucketing enable shuffle-free joins?

When two tables are bucketed on the same column with the same bucket count, Spark knows matching keys sit in the same bucket number on both sides. It joins bucket to bucket directly, with no shuffle.

PYTHON
# Bucket both tables on account_id with same count
txn.write.bucketBy(4, "account_id").sortBy("txn_date") \
    .saveAsTable("bank.txn_bucketed")

accounts = spark.createDataFrame([
    ("SAV2001", "Rajesh Kumar", "Mumbai"),
    ("SAV2002", "Priya Sharma", "Delhi"),
    ("CUR3001", "NBI Pool Account", "Chennai")
], ["account_id", "holder_name", "branch"])

accounts.write.bucketBy(4, "account_id") \
    .saveAsTable("bank.accounts_bucketed")

# Join: shuffle-free because both bucketed on account_id with 4 buckets
txn_df = spark.table("bank.txn_bucketed")
acc_df = spark.table("bank.accounts_bucketed")

joined = txn_df.join(acc_df, "account_id")
joined.explain()
DIAGRAM
== Physical Plan ==
*(3) SortMergeJoin [account_id], [account_id]
:- *(1) Sort [account_id ASC]
:  +- *(1) FileScan parquet bank.txn_bucketed
+- *(2) Sort [account_id ASC]
   +- *(2) FileScan parquet bank.accounts_bucketed
(No Exchange/Shuffle node → shuffle-free join)

Why must you use saveAsTable?

Bucketing metadata (the bucket column, bucket count, and sort column) must be registered in the catalog so later queries know the layout. Writing to a raw path with write.parquet does not register this metadata, so Spark would not know the data is bucketed.

PYTHON
# WRONG: bucketing lost because metadata not registered
txn.write.bucketBy(4, "account_id").parquet("/bank/raw-path/")
# Spark reads this as regular parquet, no bucket awareness

# CORRECT: saveAsTable registers in catalog
txn.write.bucketBy(4, "account_id").saveAsTable("bank.txn_bucketed")
# Catalog knows: 4 buckets on account_id, sorted by txn_date

# Verify catalog metadata
spark.sql("DESCRIBE EXTENDED bank.txn_bucketed").show(truncate=False)

How does bucket pruning work?

For equality filters on the bucketing column, Spark can compute which bucket the value hashes to and read only that bucket, skipping the rest.

PYTHON
# Bucket pruning: only read bucket containing SAV2001
result = spark.table("bank.txn_bucketed") \
    .filter(F.col("account_id") == "SAV2001")
result.explain()
TEXT
== Physical Plan ==
*(1) Filter (account_id = SAV2001)
+- *(1) FileScan parquet bank.txn_bucketed
     SelectedBucketsCount: 1 out of 4

How do you combine partitioning and bucketing?

Partition by a low-cardinality column for pruning, and bucket by a high-cardinality key for shuffle-free joins within each partition.

PYTHON
# Partition by branch (pruning) + bucket by account_id (join optimization)
txn.write \
    .partitionBy("branch") \
    .bucketBy(64, "account_id") \
    .sortBy("txn_date") \
    .saveAsTable("bank.txn_optimized")

# Reading: partition pruning on branch, bucket-aware join on account_id
optimized = spark.table("bank.txn_optimized")
mumbai = optimized.filter(F.col("branch") == "Mumbai")  # partition pruning
# Join on account_id is shuffle-free if accounts table matches bucket layout

What are the caveats of bucketing?

Bucketing has several requirements and limitations to be aware of.

PYTHON
# Caveat 1: Both tables must share same bucket count and column
# txn: bucketBy(4, "account_id")
# accounts: bucketBy(4, "account_id")
# Mismatch (e.g., 4 vs 8) → shuffle still happens

# Caveat 2: Must use saveAsTable, not write.parquet
# Metadata lives in catalog only

# Caveat 3: Classic Hive bucketing not supported by Delta Lake
# Delta uses Z-ordering and liquid clustering instead

# Caveat 4: Bucket count should be chosen carefully
# Too few → large files, skew possible
# Too many → small files
# Rule of thumb: aim for 100-500 MB per bucket

total_gb = 100  # 100 GB of data
target_mb = 256
buckets = int(total_gb * 1024 / target_mb)
print(f"Recommended buckets: {buckets}")
TEXT
Recommended buckets: 400

What should a senior engineer know?

A senior engineer uses partitioning and bucketing together as complementary tools: partitioning for low-cardinality filter columns (directory-level pruning), bucketing for high-cardinality join/group columns (shuffle-free operations). They verify shuffle elimination in the physical plan and match bucket counts across tables.

PYTHON
# Senior verification: confirm no shuffle in join
txn_df = spark.table("bank.txn_bucketed")
acc_df = spark.table("bank.accounts_bucketed")

plan = txn_df.join(acc_df, "account_id").explain(True)
# Look for absence of Exchange node → shuffle-free

# Senior tip: check bucket metadata
spark.sql("DESCRIBE EXTENDED bank.txn_bucketed").show(50, False)
# Confirm: Num Buckets, Bucket Columns, Sort Columns

Visual Overview

DIAGRAM
Partitioning vs Bucketing:

PARTITIONING (low cardinality)         BUCKETING (high cardinality)
  /branch=Mumbai/                       /bucket-000.parquet
    part-0.parquet                      /bucket-001.parquet
  /branch=Delhi/                        /bucket-002.parquet
    part-0.parquet                      /bucket-003.parquet
  /branch=Chennai/                      (always 4 files, any cardinality)
    part-0.parquet
  (one dir per value, variable count)

Bucket-Aware Shuffle-Free Join:
  Table A (4 buckets on account_id)    Table B (4 buckets on account_id)
  ┌──────────┐                         ┌──────────┐
  │ Bucket 0 │◄───── join ────────────►│ Bucket 0 │
  │ SAV2001  │       (no shuffle)       │ SAV2001  │
  └──────────┘                         └──────────┘
  ┌──────────┐                         ┌──────────┐
  │ Bucket 1 │◄───── join ────────────►│ Bucket 1 │
  │ SAV2002  │       (no shuffle)       │ SAV2002  │
  └──────────┘                         └──────────┘
  Matching keys in same bucket → no data movement needed

Key Takeaway

Bucketing divides a dataset into a fixed number of files by hashing a column and taking the modulo of the bucket count, so every row with the same key value always lands in the same bucket number. I declare it at write time with bucketBy, giving the number of buckets and the column, optionally adding sortBy to keep each bucket sorted, and I must use saveAsTable because the bucketing metadata has to be registered in the catalog for later queries to exploit it.

The fundamental difference from partitioning is the output shape. Partitioning creates one directory per distinct value, so the number of directories grows with the column's cardinality and is variable. Bucketing creates exactly the number of files I choose, independent of cardinality. That is why bucketing is the right tool for high-cardinality keys like account_id, where partitioning would explode into millions of directories.

The payoff is at read time for operations on the bucketing column. When two tables are bucketed on the same column with the same bucket count, a join on that column needs no shuffle, because Spark knows matching keys sit in the same bucket number on both sides and can join bucket to bucket directly. At the National Bank of India we bucketed the transactions and accounts tables by account_id into the same number of buckets, and the daily join that used to shuffle two huge tables became shuffle-free, cutting the runtime sharply.

The way I summarize it in interviews is that partitioning is for low-cardinality columns you filter on and gives pruning through directories, while bucketing is for high-cardinality columns you join or group on and gives shuffle-free operations through a fixed number of hash files.

Remember one sentence: bucketing hashes a column into a fixed number of files for shuffle-free joins, while partitioning creates value-named directories for query pruning.

---

Share
Use ← → arrow keys to navigate