Back to Advanced Data Engineering
Q12Question 12 of 24

Narrow vs wide dependencies

Key Concepts
  • A narrow dependency means each parent partition feeds at most one child partition, enabling pipelining with no shuffle.
  • A wide dependency means one parent partition feeds multiple children, forcing an expensive network shuffle and a stage boundary.
  • Consecutive narrow operations are grouped into one stage; each wide dependency starts a new stage.
  • reduceByKey is preferred over groupByKey because it combines locally before the shuffle, sending far less data.
  • Co-partitioned or bucketed datasets can turn a join from a wide dependency into a narrow one by pre-locating matching keys.
  • Data skew on a wide dependency creates straggler tasks because one shuffle partition becomes much larger than the rest.

Definition

Narrow dependencies keep each child partition fed by one parent partition with no shuffle, while wide dependencies spread one parent partition across many children and force an expensive network shuffle.
  • Narrow dependency – each parent partition is consumed by at most one child partition, as in map, filter, and flatMap, so Spark pipelines the work inside one task with no data movement.
  • Wide dependency – a single parent partition contributes records to multiple child partitions, as in groupByKey, reduceByKey, and join, which forces a shuffle that redistributes data across executors.
  • Stage boundaries – Spark groups consecutive narrow operations into one stage and starts a new stage at every wide dependency.
  • Cost implication – shuffles are the dominant cost in most Spark jobs, involving disk I/O, network transfer, and serialization.

Imagine clerks at the National Bank of India each holding one box of transaction slips. A narrow operation is like stamping every slip in your own box as processed – each clerk works within their own box, no slips leave the desk, and everyone finishes in parallel. A wide operation is like re-sorting every slip in the whole office so all of Rajesh's slips end up in one box and all of Priya's in another – slips must be carried between every desk, there is a big reshuffle, and everyone waits until it finishes before the next step can start.

Understanding the Concepts

What is a narrow dependency?

A narrow dependency is one where each parent partition feeds at most one child partition. Operations like map, filter, flatMap, and mapPartitions are narrow because every output record comes from a single input partition.

PYTHON
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("NarrowDemo").getOrCreate()

# Each partition is processed independently – no shuffle
txn = spark.read.parquet("/lake/bronze/transactions")

filtered = txn.filter(F.col("amount") > 10000)          # narrow
with_flag = filtered.withColumn(                          # narrow
    "high_value", F.lit(True)
)

Because no data moves between partitions, Spark pipelines these operations into a single task per partition, reading the data once and transforming it in a single pass.

What is a wide dependency?

A wide dependency means a single parent partition contributes records to many child partitions. Operations like groupByKey, reduceByKey, join, distinct, and repartition are wide because matching keys are scattered across all parent partitions and must be brought together.

PYTHON
# Wide: Spark must shuffle to group all records for the same account
spend_per_account = txn.groupBy("account_id").agg(
    F.sum("amount").alias("total_spend")
)

Spark performs a shuffle: each task writes its output to local disk bucketed by target partition, then the next stage reads the relevant buckets from every previous task across the network. This is expensive in disk I/O, network transfer, and serialization.

How do dependencies determine stages?

Spark groups consecutive narrow operations into one stage. Each wide dependency inserts a stage boundary because the downstream stage cannot start until all shuffle writes from the upstream stage complete.

PYTHON
# Stage 1 (narrow): read -> filter -> withColumn
# --- shuffle boundary ---
# Stage 2 (narrow after shuffle): aggregate
result = (
    txn
    .filter(F.col("branch_code") == "NBI-MUM-001")
    .withColumn("fee", F.col("amount") * 0.001)
    .groupBy("account_id")
    .agg(F.sum("fee").alias("total_fee"))
)
result.explain()
TEXT
== Physical Plan ==
*(2) HashAggregate(keys=[account_id], functions=[sum(fee)])
+- Exchange hashpartitioning(account_id, 200)   <-- shuffle
   +- *(1) HashAggregate(keys=[account_id], functions=[partial_sum(fee)])
      +- *(1) Project [account_id, (amount * 0.001) AS fee]
         +- *(1) Filter (branch_code = NBI-MUM-001)
            +- *(1) FileScan parquet

The Exchange node is the shuffle. Everything before it is Stage 1; everything after is Stage 2.

Why is reduceByKey better than groupByKey?

groupByKey ships every single value across the network before reducing. reduceByKey combines values locally on each partition first, so far less data crosses the shuffle.

PYTHON
from pyspark import SparkContext
sc = SparkContext.getOrCreate()

txn_rdd = sc.parallelize([
    ("ACC-1001", 5000), ("ACC-1002", 3000),
    ("ACC-1001", 2000), ("ACC-1002", 7000)
])

# Bad: ships all values, then reduces on the receiving side
grouped = txn_rdd.groupByKey().mapValues(sum)  # wide, heavy shuffle

# Good: combines locally first, shuffles only partial sums
reduced = txn_rdd.reduceByKey(lambda a, b: a + b)  # wide, light shuffle

With reduceByKey, each partition sends one partial sum per key instead of every value, which can reduce shuffle data by orders of magnitude on skewed keys.

How does co-partitioning turn a wide dependency into a narrow one?

If two RDDs or DataFrames are already partitioned by the same key with the same partitioner, a join between them becomes narrow because matching keys are already co-located.

PYTHON
# Both tables bucketed by account_id into 200 buckets
# Spark recognizes the co-partitioning and skips the shuffle
accounts = spark.table("accounts")    # bucketed by account_id
txn_silver = spark.table("txn_silver")  # bucketed by account_id

joined = txn_silver.join(accounts, "account_id")
joined.explain()
TEXT
== Physical Plan ==
*(3) SortMergeJoin [account_id], [account_id]
:- *(1) Sort [account_id ASC]
:  +- *(1) FileScan parquet [account_id, ...]  (no Exchange)
+- *(2) Sort [account_id ASC]
   +- *(2) FileScan parquet [account_id, ...]  (no Exchange)

No Exchange node means no shuffle – the join is narrow because the data was pre-bucketed.

How does data skew make wide dependencies worse?

Skew means one key has far more records than others. After the shuffle, its partition is enormous while the rest are small, creating a straggler task.

PYTHON
# One mega-account has 10 million transactions; others have a few thousand
skewed = txn.groupBy("account_id").agg(F.count("*").alias("cnt"))
skewed.orderBy(F.desc("cnt")).show(3)
TEXT
+----------+----------+
|account_id|       cnt|
+----------+----------+
| ACC-9999 | 10000000 |
| ACC-1001 |     5200 |
| ACC-1002 |     4800 |
+----------+----------+

The task handling ACC-9999 runs long after every other task finishes. The fix is manual salting or enabling AQE's skew join optimization.

What are common mistakes with dependencies?

The most frequent mistakes are using groupByKey when reduceByKey or a DataFrame aggregation would suffice, calling repartition unnecessarily which adds a shuffle, and not checking the physical plan for unexpected Exchange nodes.

PYTHON
# Mistake: unnecessary repartition adds an extra shuffle
bad = txn.repartition(100).groupBy("account_id").count()

# Correct: let Spark handle partitioning
good = txn.groupBy("account_id").count()

What should a senior engineer know?

A senior engineer recognizes that the number of wide dependencies equals the number of shuffle boundaries, which determines the stage count. Reducing shuffles is the single biggest performance lever in Spark.

  • Pre-partition data by the join key using bucketing to eliminate shuffle joins.
  • Use broadcast joins when one side is small to avoid shuffling the large side entirely.
  • Enable AQE so Spark can coalesce small post-shuffle partitions, split skewed partitions, and switch to broadcast joins at runtime.
  • Inspect the physical plan for Exchange nodes – each one is a shuffle.
PYTHON
# Broadcast join eliminates the shuffle of the large side entirely
from pyspark.sql.functions import broadcast

branch_ref = spark.table("branch_reference")  # small dimension

enriched = txn.join(broadcast(branch_ref), "branch_code")
enriched.explain()
TEXT
== Physical Plan ==
*(2) BroadcastHashJoin [branch_code], [branch_code]
:- *(2) FileScan parquet transactions
+- BroadcastExchange HashedRelationBroadcastMode
   +- *(1) FileScan parquet branch_reference

The large side has no Exchange – it is read once with no shuffle.

What are common interview follow-up questions?

  • How many stages does a job have? Count the wide dependencies and add one.
  • Can a join ever be narrow? Yes, if both sides are co-partitioned by the join key.
  • Does caching remove a shuffle? No. Caching avoids recomputing the lineage including the shuffle, but the first computation still pays the shuffle cost.

Visual Overview

DIAGRAM
  Narrow Dependency              Wide Dependency
  ──────────────────             ──────────────────
  Parent Partition 1             Parent Partition 1
       │                              │ ╲
       ▼                              │   ╲
  Child Partition 1              Child P1  Child P2
                                      │   ╱
  Parent Partition 2             Parent Partition 2
       │                              │ ╲
       ▼                              │   ╲
  Child Partition 2              Child P1  Child P2

  One-to-one mapping             Many-to-many mapping
  No data movement               Shuffle across network
  Same stage                     New stage boundary
DIAGRAM
  Stage Breakdown Example
  ───────────────────────
  ┌──────────────────────────────┐
  │  Stage 1 (narrow)            │
  │  read → filter → withColumn  │
  └──────────────┬───────────────┘
                 │  shuffle (Exchange)
  ┌──────────────▼───────────────┐
  │  Stage 2 (narrow)            │
  │  aggregate                   │
  └──────────────────────────────┘

Key Takeaway

The difference between narrow and wide dependencies is one of the most important concepts in Spark because it explains both how jobs are divided into stages and where the cost goes. A narrow dependency means each parent partition feeds at most one child partition, so Spark pipelines the entire chain of narrow operations inside one task on one executor with no network movement. A wide dependency means one parent partition feeds multiple children, forcing a shuffle that redistributes data across the network and creating a stage boundary.

The practical consequence is that shuffles are usually the dominant cost in a Spark job. Reducing or cheapening them is the single biggest performance lever. Replacing groupByKey with reduceByKey combines values locally before the shuffle so far less data crosses the network. Pre-partitioning data by the join key using bucketing turns shuffle joins into narrow co-located joins. Broadcast joins eliminate the large side's shuffle entirely when one side is small.

At the National Bank of India, recognizing which operations were wide and designing to avoid or shrink those shuffles was consistently the biggest performance win. Replacing groupByKey with a DataFrame aggregation for per-account totals, bucketing the transaction and account tables by account_id so their join needed no shuffle, and watching for data skew on mega-accounts that created straggler tasks – these three changes transformed our nightly pipeline from hours of shuffle-heavy work into a fast, predictable run.

Remember one sentence: narrow dependencies pipeline inside one task with no shuffle, wide dependencies force an expensive network shuffle and a new stage, so reducing shuffles is how you make Spark fast.

---

Share
Use ← → arrow keys to navigate