Performance Tuning

AQE kills join stragglers by detecting skewed partitions through a factor-of-median and byte threshold, splitting them, and replicating the other side; salting is the manual fallback for extreme or non-join skew.

On high-cardinality unsorted columns, min/max ranges span everything and dictionaries explode, so only a bloom filter's 'definitely not present' answer lets Parquet skip blocks.

Do not broadcast when the small side does not fit in executor memory, when both sides are large, or when overhead outweighs benefit: smaller than the other side is not the same as small enough to broadcast.

AQE watches real runtime sizes and upgrades a planned sort-merge join to a broadcast join when a side turns out small enough after filtering.

A broadcast variable is a general read-only object you distribute for manual map-side lookups; a broadcast join is the SQL/DataFrame optimization that broadcasts a table to avoid shuffling the other.

Bucket on high-cardinality join or group keys with a sensible count, register via saveAsTable, keep both join sides identical, and use clustering instead of bucketing on Delta.

Try AQE skew handling first because it is automatic and code-free; use salting or isolate-and-broadcast when AQE cannot cope with both-sides skew, groupBy skew, or when you need manual control.

Fight skew by isolating-and-broadcasting hot keys, increasing parallelism, pre-aggregating before the shuffle, changing the join strategy, or redesigning the model so the hot key disappears.