Definition
- coalesce(n) – a narrow transformation that merges existing partitions without a shuffle; cheap but can only decrease and produces uneven output.
- coalesce(n, shuffle=true) – inserts a full shuffle exchange before the merge, giving round-robin redistribution into exactly n balanced partitions.
- Equivalence –
repartition(n)is implemented internally ascoalesce(n, shuffle=true), so the two are identical in behaviour and cost.
Normal coalesce is like telling two neighbouring clerks at the National Bank of India 'combine your two stacks of slips into one' – no slips travel across the room. But coalesce with the shuffle flag on is like first collecting every slip into the middle of the room and re-dealing them evenly to n clerks. That re-dealing is a shuffle, and re-dealing evenly into n stacks is precisely what repartition does.
Understanding the Concepts
What is coalesce(n) without the shuffle flag?
By default, coalesce(n) avoids a shuffle. Spark marks the dependency as narrow and groups existing partitions so that several parents feed one child partition on the same executor. This is cheap but has two limits: it can only decrease the count, and the resulting partitions inherit the skew of the originals.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.appName("NBI-Coalesce-Shuffle").getOrCreate()
txn = spark.range(10000).withColumn("amount", F.rand() * 100000)
print("Start partitions:", txn.rdd.getNumPartitions())
# Default coalesce: narrow, no shuffle
coalesced = txn.coalesce(2)
print("Coalesced to:", coalesced.rdd.getNumPartitions())
# Check partition sizes: may be uneven
for i, part in enumerate(coalesced.rdd.glom().collect()):
print(f" Partition {i}: {len(part)} rows")
Start partitions: 8
Coalesced to: 2
Partition 0: 5000 rows
Partition 1: 5000 rows
What does coalesce(n, shuffle=true) do?
When you set the shuffle flag to true, Spark inserts an exchange (shuffle) that redistributes records round-robin. The code path is the same one repartition uses. The output is n evenly balanced partitions, and because of the shuffle boundary, it can also increase the partition count.
# RDD-level API: coalesce with shuffle=True
rdd = spark.sparkContext.parallelize(range(10000), 8)
# coalesce with shuffle=True: full shuffle, even output
shuffled = rdd.coalesce(4, shuffle=True)
print("Partitions:", shuffled.getNumPartitions())
for i, part in enumerate(shuffled.glom().collect()):
print(f" Partition {i}: {len(part)} rows")
Partitions: 4
Partition 0: 2500 rows
Partition 1: 2500 rows
Partition 2: 2500 rows
Partition 3: 2500 rows
Why is coalesce(n, shuffle=true) equivalent to repartition(n)?
In the Spark source code, repartition(numPartitions) calls coalesce(numPartitions, shuffle = true). The equivalence is literal, not just conceptual. Any claim you make about repartition(n) – even balance, ability to increase the count, cost of a full shuffle – applies identically to coalesce(n, shuffle=true).
# Demonstration: both produce identical plans
df = spark.range(1000)
plan_repart = df.repartition(4)._jdf.queryExecution().toString()
plan_coalesce = df.coalesce(4)._jdf.queryExecution().toString()
# repartition(4) shows Exchange (shuffle)
# coalesce(4) shows NO Exchange (narrow)
df.repartition(4).explain()
print("---")
df.coalesce(4).explain()
== Physical Plan ==
Exchange RoundRobinPartitioning(4)
+- Range (0, 1000, step=1, splits=8)
---
== Physical Plan ==
Coalesce 4
+- Range (0, 1000, step=1, splits=8)
How does the shuffle flag preserve upstream parallelism?
Plain coalesce has no shuffle boundary, so the small target partition count is pushed up the dependency chain. Any expensive computation above the coalesce is forced to run in that small number of tasks. The shuffle flag inserts a boundary that decouples the stages.
# WITHOUT shuffle flag: upstream parallelism collapses
# The enrichment runs in only 2 tasks (bad)
result_bad = txn \
.withColumn("tax", F.col("amount") * 0.18) \
.withColumn("net", F.col("amount") - F.col("tax")) \
.coalesce(2) # narrow: pushes 2-task parallelism upstream
# WITH shuffle flag (or repartition): upstream keeps full parallelism
# The enrichment runs in 8 tasks, then shuffles to 2 (good)
result_good = txn \
.withColumn("tax", F.col("amount") * 0.18) \
.withColumn("net", F.col("amount") - F.col("tax")) \
.repartition(2) # shuffle boundary preserves upstream parallelism
result_good.write.parquet("/bank/reports/net-amounts/")
When should you use coalesce(n, shuffle=true) vs repartition(n)?
They are the same operation. The only reason to write coalesce(n, shuffle=true) instead of repartition(n) is historical or for explicitness in an existing coalesce-based code path. In practice, just use repartition(n) for clarity.
# These two are identical:
df.repartition(4)
# is the same as:
df.coalesce(4) # This is NOT the same (no shuffle)
# To get the equivalent:
rdd.coalesce(4, shuffle=True) # RDD API: same as repartition
# In DataFrame API, there is no shuffle flag, so just use repartition
df.repartition(4) # preferred for clarity
What happens when coalesce reduces a large number to a small number?
When you coalesce from a very large number to a very small number without the shuffle flag, the narrow dependency forces the entire upstream computation into the small number of tasks. This is the most dangerous pitfall.
# Dangerous: 200 partitions coalesced to 1
# The ENTIRE pipeline runs in 1 task
result = spark.read.parquet("hdfs:///bank/transactions/2026/") \
.filter(F.col("status") == "COMPLETED") \
.groupBy("branch").agg(F.sum("amount")) \
.coalesce(1) # EVERYTHING above runs in 1 task
# Safe: repartition(1) keeps upstream parallel
result = spark.read.parquet("hdfs:///bank/transactions/2026/") \
.filter(F.col("status") == "COMPLETED") \
.groupBy("branch").agg(F.sum("amount")) \
.repartition(1) # upstream in 200 tasks, gather to 1
result.write.csv("/bank/reports/annual-summary.csv")
What are common mistakes?
The most common mistake is assuming coalesce with no shuffle flag preserves upstream parallelism. It does not. The second mistake is writing coalesce(n) and expecting it to increase partitions.
# WRONG: coalesce(1) serializes upstream heavy work
heavy = spark.read.parquet("hdfs:///bank/loans/") \
.withColumn("emi", F.col("principal") * F.col("rate") / 12) \
.coalesce(1) # entire EMI calculation in 1 task
# CORRECT: repartition(1) or coalesce after the heavy work
heavy = spark.read.parquet("hdfs:///bank/loans/") \
.withColumn("emi", F.col("principal") * F.col("rate") / 12) \
.repartition(1) # EMI runs in many tasks, then gathers
# WRONG: expecting coalesce to increase
df = spark.range(10).coalesce(1)
df.coalesce(10).rdd.getNumPartitions() # still 1!
1
What should a senior engineer know?
A senior engineer understands the internal equivalence and uses it to explain the upstream-parallelism problem. They know that the shuffle boundary is the key mechanism: with it, stages are decoupled; without it, the narrow dependency pushes the partition count upstream. They always prefer repartition(n) in the DataFrame API for clarity.
# Senior pattern: verify plans to confirm shuffle boundaries
df = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
# Check: does coalesce insert a shuffle?
df.coalesce(2).explain()
# No Exchange node → narrow, upstream parallelism at risk
# Check: does repartition insert a shuffle?
df.repartition(2).explain()
# Exchange RoundRobinPartitioning(2) → shuffle boundary present
== Physical Plan (coalesce) ==
Coalesce 2
+- FileScan parquet
== Physical Plan (repartition) ==
Exchange RoundRobinPartitioning(2)
+- FileScan parquet
Visual Overview
coalesce(n) vs coalesce(n, shuffle=true) vs repartition(n) coalesce(n) – NARROW, no shuffle: ┌─────┐ │ P0 │───┐ └─────┘ │ merge on ┌─────┐ ├──→ ┌─────┐ │ P1 │───┘ │ P0' │ (uneven) └─────┘ └─────┘ ┌─────┐ ┌──→ ┌─────┐ │ P2 │───┤ │ P1' │ (uneven) └─────┘ │ └─────┘ ┌─────┐ │ │ P3 │───┘ └─────┘ No network movement. Upstream parallelism COLLAPSES. coalesce(n, shuffle=true) = repartition(n) – WIDE, full shuffle: Stage 1 (parallel) Shuffle Stage 2 ┌─────┐ Exchange ┌─────┐ │ P0 │──┐ ┌│ P0' │ (even) └─────┘ │ round-robin ┌────────┐ │└─────┘ ┌─────┐ ├──────────────→│Shuffle │────→┤┌─────┐ │ P1 │──┤ │Exchange│ ││ P1' │ (even) └─────┘ │ └────────┘ │└─────┘ ┌─────┐ │ │ │ P2 │──┤ (all tasks │ └─────┘ │ run in │ ┌─────┐ │ parallel) │ │ P3 │──┘ │ └─────┘ │ Upstream parallelism PRESERVED by shuffle boundary.
Key Takeaway
By default, coalesce is a narrow transformation that reduces partitions without a shuffle by stitching existing partitions together on the same executor. It is cheap, but it can only decrease the count and it leaves partitions uneven. The optional shuffle flag changes everything: when it is true, Spark inserts a full shuffle exchange and redistributes records round-robin into exactly the number of partitions requested. That is identical to repartition, and the equivalence is literal: in the Spark source, repartition with a number simply calls coalesce with that number and the shuffle flag set to true.
The practical reason the shuffle flag matters is upstream parallelism. Plain coalesce has no shuffle boundary, so the small target partition count is pushed up the dependency chain, and any expensive computation above the coalesce is forced to run in that small number of tasks. Setting the shuffle flag inserts a boundary that decouples the stages, so the expensive work keeps its full parallelism and only the final stage runs with the reduced count.
In my settlement job at the National Bank of India I hit exactly this. We read two hundred partitions of trades, ran an expensive enrichment, and wanted four output files. Writing plain coalesce of four collapsed the enrichment into four tasks and the job crawled. Switching to coalesce of four with the shuffle flag (which is just repartition of four) kept the enrichment running across two hundred tasks and only reduced to four at write time. The runtime dropped from hours to minutes.
So my rule is that coalesce without shuffle is the cheap merge for when upstream parallelism does not matter, and coalesce with the shuffle flag is just another name for repartition, which I use when I need a shuffle boundary or even output.
Remember one sentence: coalesce(n, shuffle=true) is literally repartition(n), and the shuffle boundary is what preserves upstream parallelism that plain coalesce would collapse.
---