Definition
- What it does – walks a JVM object graph and estimates the total byte footprint including object headers, field layout, alignment padding, and references.
- Why it matters – in-memory size is larger than on-disk size; a 50 MB Parquet file can be 300+ MB as live JVM objects.
- Main use – deciding whether a table is small enough to broadcast for a broadcast join without causing OOM.
- Access – in Scala:
SizeEstimator.estimate(obj); in PySpark:spark._jvm.org.apache.spark.util.SizeEstimator.estimate(df._jdf).
The National Bank of India wants to mail a copy of a branch reference list to every single teller in the country so they do not have to phone head office for each lookup. Before mailing copies to thousands of tellers, you need to know how heavy one copy is, because if each is a thick five-kilogram binder, mailing thousands crushes the postroom. SizeEstimator is the scale that weighs one copy as it actually sits in memory, so you decide whether it is light enough to send everywhere.
Understanding the Concepts
Why is in-memory size different from on-disk size?
JVM objects carry overhead beyond the raw data: headers (12-16 bytes per object), references (4-8 bytes each), alignment padding, and boxed wrappers for primitives.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("SizeEstDemo").getOrCreate()
# Branch reference table: 50 MB on disk as compressed Parquet
branch_ref = spark.read.parquet("/lake/dim/branches")
print(f"Row count: {branch_ref.count()}")
# On disk: 50 MB
# In memory as JVM objects: 300+ MB (6x expansion)
Guessing from file size or row count is unreliable. That is why SizeEstimator exists.
How do you use SizeEstimator?
In PySpark, access the JVM utility through the gateway.
# Estimate the in-memory size of the DataFrame's internal representation
size_bytes = spark._jvm.org.apache.spark.util.SizeEstimator.estimate(
branch_ref._jdf
)
size_mb = size_bytes / (1024 * 1024)
print(f"Estimated in-memory size: {size_mb:.1f} MB")
Estimated in-memory size: 312.5 MB
In Scala, call it directly:
# Scala equivalent:
# import org.apache.spark.util.SizeEstimator
# val sizeBytes = SizeEstimator.estimate(branchRef)
Why does this matter for broadcast joins?
A broadcast join sends a full copy of the small table to every executor. If the in-memory size exceeds executor memory, you get OOM.
# Spark auto-broadcasts below the threshold (default 10 MB)
print(spark.conf.get("spark.sql.autoBroadcastJoinThreshold")) # 10485760
# But the 10 MB threshold is based on logical/compressed size,
# not the inflated in-memory size
# Dangerous: force broadcast of something too large in memory
from pyspark.sql.functions import broadcast
result = txn.join(broadcast(branch_ref), "branch_code") # OOM risk!
What is the safe broadcast decision pattern?
Sample, estimate, multiply, and add a margin.
# Step 1: Get a sample to the Driver
sample_rows = branch_ref.take(100)
# Step 2: Estimate per-row size
sample_df = spark.createDataFrame(sample_rows, branch_ref.schema)
sample_size = spark._jvm.org.apache.spark.util.SizeEstimator.estimate(
sample_df._jdf
)
per_row_bytes = sample_size / len(sample_rows)
# Step 3: Multiply by total row count
total_rows = branch_ref.count()
estimated_total = per_row_bytes * total_rows
# Step 4: Add safety margin (1.5x)
safe_estimate = estimated_total * 1.5
# Step 5: Compare against a broadcast ceiling
broadcast_ceiling = 500 * 1024 * 1024 # 500 MB max
if safe_estimate < broadcast_ceiling:
print(f"Safe to broadcast: {safe_estimate / (1024*1024):.0f} MB")
result = txn.join(broadcast(branch_ref), "branch_code")
else:
print(f"Too large to broadcast: {safe_estimate / (1024*1024):.0f} MB")
result = txn.join(branch_ref, "branch_code") # shuffle join
How do you check cached size instead?
Another approach is to cache the DataFrame and read the actual size from the Spark UI Storage tab.
branch_ref.cache()
branch_ref.count() # materialize the cache
# Check Spark UI -> Storage tab:
# Size in Memory: 312.5 MB
# Size on Disk: 0 B (MEMORY_ONLY)
# Or programmatically:
catalog = spark._jvm.org.apache.spark.storage.StorageLevel
# Use the Spark UI REST API: /api/v1/applications/{appId}/storage/rdd
What are common mistakes?
The most frequent mistake is trusting the on-disk size or the autoBroadcastJoinThreshold without checking in-memory inflation.
# Mistake: "it's only 50 MB on disk, broadcast is fine"
result = txn.join(broadcast(branch_ref), "branch_code") # OOM!
# Fix: estimate the real in-memory size first
# The 50 MB Parquet file is 300+ MB in memory
# With safety margin: 450+ MB per executor
What should a senior engineer know?
- SizeEstimator is an estimate, not exact. It can be off for arrays of primitives or deeply nested structures. Always add a safety margin.
- The estimate is for the JVM object graph, not for Tungsten UnsafeRow format. DataFrames use UnsafeRow internally, which is more compact.
- For the most accurate size, cache the DataFrame and read the Storage tab.
- Set the broadcast threshold conservatively: well below executor memory, not at the limit.
- In production, build a small utility function that estimates and decides automatically.
def safe_broadcast_join(large_df, small_df, join_col, max_mb=500):
"""Join with broadcast only if small_df is safely below max_mb."""
sample = small_df.take(100)
sample_df = spark.createDataFrame(sample, small_df.schema)
per_row = (spark._jvm.org.apache.spark.util.SizeEstimator
.estimate(sample_df._jdf) / len(sample))
estimated_mb = (per_row * small_df.count() * 1.5) / (1024 * 1024)
if estimated_mb < max_mb:
return large_df.join(broadcast(small_df), join_col)
else:
return large_df.join(small_df, join_col)
What are common interview follow-up questions?
- Does SizeEstimator measure off-heap memory? No, it estimates JVM heap object graph size.
- Can you use it on an RDD? Yes, estimate any JVM object, including RDDs.
- What is the default broadcast threshold? 10 MB (logical size), configurable via spark.sql.autoBroadcastJoinThreshold.
Visual Overview
Broadcast Join Decision
────────────────────────
On-disk size: 50 MB (compressed Parquet)
│
│ SizeEstimator
▼
In-memory size: 312 MB (JVM objects with headers/padding)
│
│ × 1.5 safety margin
▼
Safe estimate: 468 MB
│
├── < 500 MB ceiling? → YES → Broadcast Join (fast)
│
└── > 500 MB ceiling? → NO → Shuffle Join (safe)
Never trust on-disk size alone for broadcast decisions!
Key Takeaway
SizeEstimator is a Spark utility that estimates how many bytes a JVM object and its references occupy in heap memory, accounting for headers, padding, and boxed values. The reason it matters is that in-memory size is almost always larger than on-disk size – a 50 MB Parquet file can inflate to 300+ MB as live JVM objects. Guessing from file size or row count is unreliable.
The main use is deciding whether to broadcast. A broadcast join sends a full copy of the small table to every executor, which is a huge performance win, but only if the table genuinely fits in memory. Spark auto-broadcasts below a 10 MB threshold, but those statistics can be missing or wrong. Forcing a broadcast of something that is actually hundreds of megabytes in memory blows the Driver or executors.
At the National Bank of India, someone forced a broadcast of an account reference table because it was only a few megabytes on disk. In production, the executors ran out of memory because the expanded JVM objects were many times larger. After that, we built a small utility that samples rows, estimates per-row in-memory bytes, multiplies by the row count, adds a safety margin, and only broadcasts when the result is safely below a ceiling. That turned broadcasting from a risky guess into a safe, repeatable decision.
Remember one sentence: use SizeEstimator to measure the real in-memory size before broadcasting, because on-disk size and row count are unreliable guides for JVM memory consumption.
---