Data skew in Spark: hot keys and how to detect it

• Data skew – uneven distribution of data across partitions after a shuffle, where one or a few partitions hold far more records than the rest.
• Hot key – a join or group key that appears disproportionately often, funneling most rows into a single partition.
• Straggler task – the one task processing the oversized partition that runs much longer than all its peers, delaying the whole stage.
• Detection – in the Spark UI Stages tab, the maximum task duration is ten to a hundred times the median, and the straggler's shuffle read is far larger than the others.
• Root cause – any wide operation (join, groupBy, repartition) that hashes data by a skewed key creates the imbalance.

What is data skew in Spark?

Data skew occurs when data is distributed unevenly across partitions after a shuffle. In Spark, wide operations like join, groupBy, and repartition hash rows by key so that all rows sharing the same key land in the same partition. If one key appears far more often than the others, its partition becomes enormous while the rest stay small. Since Spark runs exactly one task per partition, the oversized partition gets a single task that must process all those rows alone.

# Simulate skewed transaction data at the National Bank of India
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, lit, rand

spark = SparkSession.builder.appName("SkewDemo").getOrCreate()

# 10 million rows for one corporate account, 1000 rows each for 999 others
skewed_txns = (
    spark.range(10000000).withColumn("account_id", lit("CORP_POOL_001"))
    .union(
        spark.range(999000).withColumn("account_id",
            (col("id") % 999 + 1).cast("string"))
    )
    .withColumn("amount", (rand() * 100000).cast("decimal(12,2)"))
)

# This groupBy triggers a shuffle; one partition gets 10M rows
account_totals = skewed_txns.groupBy("account_id").sum("amount")
account_totals.explain()

Why does data skew happen?

Skew arises from the nature of real-world data. In banking, certain accounts generate far more activity than others. A corporate pooling account at National Bank of India might handle millions of daily transactions while a personal savings account handles a handful. When Spark hashes the account id to decide which partition receives each row, the pooling account's rows all land in one partition.

Common sources of skew in banking pipelines include:

• A corporate sweep or pooling account with millions of daily transactions.
• A suspense account that collects all unmatched or error transactions.
• A null or default placeholder id that every row without a real key carries.
• A popular merchant id in a payments dataset that processes high volumes.

# Identify hot keys by counting rows per key
hot_keys = (
    skewed_txns.groupBy("account_id")
    .count()
    .orderBy(col("count").desc())
)
hot_keys.show(5)

+-------------+--------+
|   account_id|   count|
+-------------+--------+
|CORP_POOL_001|10000000|
|          500|    1000|
|          250|    1000|
|          750|    1000|
|          125|    1000|
+-------------+--------+

How do you detect data skew in the Spark UI?

The Spark UI Stages tab is the primary detection tool. Open the stage that follows a shuffle and examine the task summary. With skew you will see:

• The maximum task duration is ten to a hundred times the median task duration.
• The maximum shuffle read size is far larger than the median shuffle read.
• The event timeline shows most tasks finishing early and one task running alone for a long time.
• The straggler task may show spill to disk because its partition does not fit in memory.

# After running account_totals.count(), open the Spark UI at localhost:4040
# Navigate to Stages tab -> click the groupBy stage
# Check the Summary Metrics table:
#   Metric          Min     25th    Median    75th     Max
#   Duration        0.1s    0.2s    0.3s      0.4s     45.0s   <-- skew!
#   Shuffle Read    10 KB   15 KB   20 KB     25 KB    1.2 GB  <-- skew!

How is data skew different from too few partitions?

These two problems look similar (slow stages) but have different signatures and different fixes.

| Characteristic | Data Skew | Too Few Partitions |
|---|---|---|
| Task duration spread | One task is 10-100x the median | All tasks are roughly equally slow |
| Shuffle read spread | One task has a far larger read | All tasks have similar large reads |
| Root cause | Uneven key distribution | Partition count is too low |
| Fix | Salting, AQE skew join, isolate hot key | Increase spark.sql.shuffle.partitions |

# Too few partitions: all tasks are equally large
spark.conf.set("spark.sql.shuffle.partitions", "2")
even_data = spark.range(10000000).withColumn("account_id",
    (col("id") % 1000).cast("string"))
even_data.groupBy("account_id").count().count()
# UI shows 2 tasks, both slow but roughly equal duration

# Data skew: one task is far larger
spark.conf.set("spark.sql.shuffle.partitions", "200")
skewed_txns.groupBy("account_id").sum("amount").count()
# UI shows 200 tasks, 199 fast and 1 extremely slow

What are the consequences of data skew?

The consequences go beyond just slowness. A single straggler task can cause cascading problems in a production banking pipeline.

• Stage delay – the entire stage waits for the slowest task, so the job runtime equals the straggler runtime regardless of how fast the other tasks finished.
• Resource waste – executor cores assigned to completed tasks sit idle waiting for the straggler, wasting cluster capacity.
• Spill to disk – the oversized partition may not fit in the task's memory budget, forcing Spark to spill intermediate data to disk, which slows the task even further.
• Out-of-memory failure – if the partition is large enough it can exceed the executor's memory entirely, killing the task and potentially the executor.
• Downstream delays – in an orchestrated pipeline at the National Bank of India, a skewed stage delays every downstream job that depends on its output.

# Demonstrate spill: the straggler task spills to disk
# In the Spark UI Stages tab, check the "Shuffle Spill (Memory)" and
# "Shuffle Spill (Disk)" columns for the straggler task
# A large spill-to-disk value confirms the partition exceeded memory

What are common fixes for data skew?

Several strategies address skew, and the right choice depends on the operation and the data.

• Adaptive Query Execution (AQE) skew join – Spark 3.0+ can detect skewed partitions at runtime and split them automatically. Enable with spark.sql.adaptive.enabled=true and spark.sql.adaptive.skewJoin.enabled=true.
• Salting – append a random suffix to the hot key to spread it across multiple partitions, and replicate the other side to match.
• Isolate and broadcast – filter out the hot key, broadcast-join it separately, and union the results.
• Pre-aggregation – for groupBy skew, use reduceByKey or partial aggregation to shrink the data before the shuffle.

from pyspark.sql.functions import when, floor, broadcast

# AQE skew join: enable and let Spark handle it automatically
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes",
    "256MB")

# Isolate-and-broadcast approach for a known hot key
hot_txns = skewed_txns.filter(col("account_id") == "CORP_POOL_001")
cold_txns = skewed_txns.filter(col("account_id") != "CORP_POOL_001")

accounts = spark.read.parquet("/data/accounts")

cold_result = cold_txns.join(accounts, "account_id")
hot_result = hot_txns.join(broadcast(accounts), "account_id")
final_result = cold_result.union(hot_result)

How do you monitor for emerging skew over time?

Skew can appear gradually as data patterns change. A personal account that becomes a corporate pooling account, or a new high-volume merchant, can introduce skew that did not exist when the pipeline was first built.

• Monitor task-duration ratios over time using Spark event logs and the History Server.
• Set alerts when the max-to-median task duration ratio exceeds a threshold (for example, ten times).
• Schedule periodic key-distribution profiling jobs that count rows per key and flag new hot keys.
• Watch for skew in writes too, where a partitionBy on a skewed column produces one gigantic output file.

# Periodic key distribution profiling job
from pyspark.sql.functions import count, desc

def profile_key_distribution(df, key_col, threshold_ratio=100):
    stats = df.groupBy(key_col).agg(count("*").alias("row_count"))
    total_keys = stats.count()
    avg_count = df.count() / total_keys
    hot_keys = stats.filter(col("row_count") > avg_count * threshold_ratio)
    return hot_keys.orderBy(desc("row_count"))

hot = profile_key_distribution(skewed_txns, "account_id")
hot.show()

What about skew in writes?

Skew affects writes as well as reads. When you call df.write.partitionBy("account_id").parquet(path), a skewed account id produces one enormous output file for the hot key's partition directory. This creates the small-files problem in reverse: one giant file that a single downstream task must read entirely.

# Skewed write: one partition directory gets a huge file
skewed_txns.write.partitionBy("account_id").parquet("/output/skewed_write")
# The account_id=CORP_POOL_001 directory will contain a single multi-GB file

# Fix: repartition within each partition to cap file size
skewed_txns.repartition(10, "account_id").write \
    .option("maxRecordsPerFile", 1000000) \
    .partitionBy("account_id").parquet("/output/balanced_write")

What should a senior engineer know about data skew?

• AQE is the first line of defense – always enable Adaptive Query Execution in Spark 3.0+ before reaching for manual fixes.
• Null keys are silent killers – a null join key collects every unmatched row into one partition; filter or replace nulls before joining.
• Skew can hide until data grows – a key that was fine at low volume can become a hot key as the business scales.
• Salting is not free – it duplicates the small side by the salt factor, so use targeted salting on known hot keys rather than salting everything.
• Measure before fixing – profile the key distribution and read the Spark UI before applying skew fixes, because the wrong fix wastes effort.