Definition
- repartition(n) – a wide transformation that performs a full shuffle, redistributing all records round-robin into exactly n evenly-sized partitions; can increase or decrease the count.
- coalesce(n) – a narrow transformation that stitches existing partitions together on the same executor without a shuffle; can only decrease the count and produces unevenly-sized output.
- Cost trade-off – repartition pays for a shuffle but guarantees balanced output; coalesce avoids the shuffle but inherits the skew of the original partitions.
Imagine transaction files split across clerks' desks at the National Bank of India. Repartition is dumping every file into the middle of the room and redealing them evenly into exactly the number of new piles you want – fair and balanced, but everyone has to carry their files across the room, which is slow. Coalesce is just asking a few clerks to slide their existing piles together into fewer stacks without anyone walking anywhere – fast and cheap, but the resulting stacks are whatever sizes the original piles happened to be, so they can be lopsided, and you can only ever end up with fewer stacks, never more.
Understanding the Concepts
What is repartition(n)?
Repartition is a wide transformation that triggers a full shuffle. Every record is sent across the network and distributed round-robin into the requested number of partitions. The output is evenly balanced regardless of how skewed the input was. Because it is a true shuffle, repartition can increase, decrease, or keep the partition count the same while rebalancing.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.appName("NBI-Repartition").getOrCreate()
# Simulating skewed transaction data
txn = spark.createDataFrame([
("SAV2001", 50000.00), ("SAV2001", 75000.00),
("SAV2002", 12000.00), ("CUR3001", 300000.00),
("SAV2001", 25000.00), ("SAV2002", 8000.00)
], ["account_id", "amount"])
print("Before repartition:", txn.rdd.getNumPartitions())
# Repartition into 4 even partitions (full shuffle)
repart = txn.repartition(4)
print("After repartition:", repart.rdd.getNumPartitions())
# Check partition sizes: they are even
for i, part in enumerate(repart.rdd.glom().collect()):
print(f" Partition {i}: {len(part)} rows")
Before repartition: 2
After repartition: 4
Partition 0: 2 rows
Partition 1: 1 rows
Partition 2: 2 rows
Partition 3: 1 rows
What is coalesce(n)?
Coalesce is a narrow transformation when reducing partitions. It logically combines existing partitions by assigning several parent partitions to each resulting partition on the same executor. Data does not move across the network. The trade-offs follow directly: coalesce can only decrease, and the output can be uneven because it stitches together whatever partitions existed.
# Start with 8 partitions after a filter that left most empty
txn_filtered = txn.repartition(8).filter(F.col("amount") > 50000)
print("After filter, partitions:", txn_filtered.rdd.getNumPartitions())
# Coalesce to 2 partitions: cheap, no shuffle
coalesced = txn_filtered.coalesce(2)
print("After coalesce:", coalesced.rdd.getNumPartitions())
# Output may be uneven
for i, part in enumerate(coalesced.rdd.glom().collect()):
print(f" Partition {i}: {len(part)} rows")
After filter, partitions: 8
After coalesce: 2
Partition 0: 1 rows
Partition 1: 2 rows
How is repartition different from coalesce?
| Feature | repartition(n) | coalesce(n) |
|---|---|---|
| Dependency type | Wide (shuffle) | Narrow (no shuffle) |
| Network cost | Full shuffle across executors | No data movement |
| Direction | Increase or decrease | Decrease only |
| Output balance | Evenly sized partitions | Unevenly sized (inherits input skew) |
| Use case | Rebalance, increase parallelism | Reduce partitions cheaply |
| Implementation | coalesce(n, shuffle=true) | coalesce(n, shuffle=false) |
# Side-by-side comparison
base = spark.range(1000).repartition(10)
print("Base partitions:", base.rdd.getNumPartitions())
# Repartition: can increase
more = base.repartition(20)
print("Repartition to 20:", more.rdd.getNumPartitions()) # 20
# Coalesce: cannot increase (silently stays the same)
still_ten = base.coalesce(20)
print("Coalesce to 20:", still_ten.rdd.getNumPartitions()) # still 10
# Coalesce: can decrease
fewer = base.coalesce(3)
print("Coalesce to 3:", fewer.rdd.getNumPartitions()) # 3
Base partitions: 10
Repartition to 20: 20
Coalesce to 20: 10
Coalesce to 3: 3
When should you use repartition?
Use repartition when you need balanced partitions or need to increase the partition count.
- Before a heavy parallel stage to ensure every task has equal work.
- To increase parallelism on a large dataset.
- To rebalance skewed data after a join or filter.
- To fix partition imbalance that causes straggler tasks.
# Scenario: rebalance before a heavy computation
raw_txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
print("Raw partitions:", raw_txn.rdd.getNumPartitions()) # 200, but skewed
# Rebalance into 64 even partitions before fraud scoring
balanced = raw_txn.repartition(64)
scored = balanced.withColumn(
"fraud_score",
F.when(F.col("amount") > 500000, F.lit(0.85)).otherwise(F.lit(0.10))
)
scored.write.parquet("hdfs:///bank/fraud-scores/2026-01/")
When should you use coalesce?
Use coalesce when you only need to reduce the partition count and balance does not matter.
- Before writing output to avoid a flood of tiny files.
- After a heavy filter that left many empty partitions.
- When the data is already small and no heavy computation follows.
# Scenario: reduce partitions before writing to avoid small files
alerts = scored.filter(F.col("fraud_score") > 0.80)
print("Alert partitions:", alerts.rdd.getNumPartitions()) # 64, mostly empty
# Coalesce to 2 files: cheap, no shuffle
alerts.coalesce(2).write.mode("overwrite").parquet(
"hdfs:///bank/fraud-alerts/2026-01/"
)
print("Written 2 output files")
Alert partitions: 64
Written 2 output files
What is the repartition(1) vs coalesce(1) trade-off?
Both end with a single partition, but the path differs. coalesce(1) is narrow, so the single final task also runs every upstream transformation in the same chain, squeezing all preceding work into one task with no parallelism. repartition(1) inserts a shuffle boundary, so the upstream work runs in full parallelism and only the final gather is single-threaded.
# coalesce(1): upstream work is serialized into one task
# BAD for heavy upstream computation
result_bad = raw_txn.filter(F.col("status") == "COMPLETED") \
.withColumn("tax", F.col("amount") * 0.18) \
.coalesce(1) # forces the entire pipeline into 1 task
# repartition(1): upstream runs in parallel, then gathers
# BETTER for heavy upstream computation
result_good = raw_txn.filter(F.col("status") == "COMPLETED") \
.withColumn("tax", F.col("amount") * 0.18) \
.repartition(1) # shuffle boundary keeps upstream parallel
result_good.write.csv("hdfs:///bank/reports/tax-summary.csv")
What are common mistakes with repartition and coalesce?
The most common mistake is using coalesce when upstream work is heavy, which serializes the computation. Another is coalescing to a larger number and expecting more partitions.
# WRONG: coalesce(1) before heavy computation
# This forces the entire groupBy + agg into 1 task
bad = raw_txn.coalesce(1).groupBy("branch").agg(F.sum("amount"))
# CORRECT: do the heavy work first, then coalesce
good = raw_txn.groupBy("branch").agg(F.sum("amount")).coalesce(1)
# WRONG: expecting coalesce to increase partitions
small = spark.range(10).coalesce(1)
print(small.rdd.getNumPartitions()) # 1
bigger = small.coalesce(10)
print(bigger.rdd.getNumPartitions()) # still 1, not 10!
# CORRECT: use repartition to increase
bigger = small.repartition(10)
print(bigger.rdd.getNumPartitions()) # 10
1
1
10
What should a senior engineer know?
A senior engineer knows that repartition(n) is implemented internally as coalesce(n, shuffle=true). They also know that coalesce with a narrow dependency pushes the small partition count up the chain, reducing upstream parallelism, and they choose between repartition and coalesce based on whether the upstream work is heavy and whether balance matters.
# Senior pattern: check partition sizes before deciding
def show_partition_stats(df, label):
sizes = df.rdd.mapPartitions(lambda it: [sum(1 for _ in it)]).collect()
print(f"{label}: partitions={len(sizes)}, "
f"min={min(sizes)}, max={max(sizes)}, "
f"avg={sum(sizes)/len(sizes):.1f}")
show_partition_stats(raw_txn, "Raw")
show_partition_stats(raw_txn.repartition(32), "After repartition(32)")
show_partition_stats(raw_txn.coalesce(4), "After coalesce(4)")
Raw: partitions=200, min=0, max=15000, avg=625.0
After repartition(32): partitions=32, min=3900, max=3910, avg=3906.2
After coalesce(4): partitions=4, min=8000, max=42000, avg=31250.0
Visual Overview
repartition(n) vs coalesce(n) REPARTITION (wide, full shuffle): Input (skewed) Shuffle Output (balanced) ┌──────────┐ ┌─────┐ │ P0: 8000 │──┐ round-robin ┌──→ │ P0 │ ~3333 └──────────┘ │ redistribution │ └─────┘ ┌──────────┐ ├──→ ═══════════ ──→─┤ ┌─────┐ │ P1: 100 │──┤ (network I/O) ├──→ │ P1 │ ~3333 └──────────┘ │ │ └─────┘ ┌──────────┐ │ │ ┌─────┐ │ P2: 1900 │──┘ └──→ │ P2 │ ~3334 └──────────┘ └─────┘ Total: 10000 Total: 10000 (even) COALESCE (narrow, no shuffle): Input (skewed) Stitch Output (uneven) ┌──────────┐ ┌──────────┐ │ P0: 8000 │─────────────────────────→ │ P0: 8000 │ └──────────┘ └──────────┘ ┌──────────┐ merge on ┌──────────┐ │ P1: 100 │───┐ same executor ──→ │ P1: 2000 │ └──────────┘ │ (no network) └──────────┘ ┌──────────┐ │ │ P2: 1900 │───┘ └──────────┘ Total: 10000 Total: 10000 (uneven)
Key Takeaway
Repartition and coalesce both change the number of partitions, but they work in opposite ways and I choose between them based on cost versus balance. Repartition is a wide transformation, so it performs a full shuffle: every record is sent across the network and dealt round-robin into exactly the number of partitions I ask for, which means the output is perfectly even regardless of how skewed the input was, and because it is a real shuffle it can increase the partition count, decrease it, or rebalance while keeping the count the same. The price is the shuffle itself: network transfer, disk writes, and serialization.
Coalesce, when reducing partitions, is a narrow transformation, so it does no shuffle. Instead of redistributing records, it stitches existing partitions together on the same executors, which makes it very cheap. The trade-offs follow directly from having no shuffle: coalesce can only decrease the partition count, because without moving data it cannot create new partitions, and the output can be uneven, because it just merges whatever partitions existed. Coalescing to a larger number than the current count has no effect unless you pass the shuffle flag, which turns it back into a repartition.
In my project at the National Bank of India I used each for its purpose. After filtering a large daily slice down to a small result spread across two hundred tiny partitions, I used coalesce to reduce it to a handful cheaply before writing the output files, since paying for a shuffle there would be wasteful. Before a heavy join on a skewed dataset, I used repartition to produce balanced partitions so every task had equal work. The one caution I always add is that coalescing to very few partitions also shrinks the parallelism of the stage producing the data, so if I coalesce to one before an expensive computation that whole computation runs in a single task, and in that case I coalesce after the heavy work or use repartition.
Remember one sentence: repartition shuffles for balanced output and can increase or decrease, while coalesce merges cheaply with no shuffle but only decreases and stays uneven.
---