Why "coalesce to reduce, repartition to increase": and the `repartition(1)` vs `coalesce(1)` trade-off | Data Engineering

Q: What are the risks of a single partition?

Neither singlepartition option is good for large data. One partition means one task holds everything, risking outofmemory. For writing a single file over large data, the better pattern is to keep parallelism and let the file system or a downstream step merge, or to accept multiple files. BAD: single

Definition

Coalesce merges cheaply but only shrinks; repartition shuffles to both grow and balance – so reduce with coalesce, increase with repartition.

Coalesce – narrow transformation that merges existing partitions without a shuffle; can only reduce the count and produces uneven output.
Repartition – wide transformation that performs a full shuffle; can increase or decrease and always produces balanced partitions.
Guideline – for a simple reduction where balance does not matter, coalesce avoids network cost; to increase parallelism or force even sizes, repartition is the only tool.
Single-partition trap – coalesce(1) serializes the entire upstream chain; repartition(1) preserves upstream parallelism via the shuffle boundary.

At the National Bank of India, reducing the number of clerks handling slips is like merging two neighbouring stacks – no slip has to walk across the room, so you just coalesce. But increasing the number of clerks means some slips genuinely have to move to new desks, and you cannot create new desks without moving slips, so you must repartition. Collapsing everything to one clerk is a special case: coalesce makes that one clerk also do all the upstream sorting alone, while repartition first redistributes then collapses.

Understanding the Concepts

Why is the guideline 'coalesce to reduce, repartition to increase'?

The guideline follows from the mechanics of each operation. Coalesce is narrow, so it cannot move data across executors. Without moving data, you cannot split one partition into two, so coalesce cannot increase the count. If you ask coalesce for more partitions than you have, it quietly does nothing. Repartition always shuffles, and a shuffle can create any number of new partitions.

PYTHON

from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder.appName("NBI-Coalesce-vs-Repart").getOrCreate()

df = spark.range(10000).repartition(10)
print("Start:", df.rdd.getNumPartitions())

# Coalesce can reduce
print("Coalesce to 3:", df.coalesce(3).rdd.getNumPartitions())   # 3

# Coalesce cannot increase
print("Coalesce to 20:", df.coalesce(20).rdd.getNumPartitions())  # still 10

# Repartition can do both
print("Repartition to 3:", df.repartition(3).rdd.getNumPartitions())   # 3
print("Repartition to 20:", df.repartition(20).rdd.getNumPartitions()) # 20

TEXT

Start: 10
Coalesce to 3: 3
Coalesce to 20: 10
Repartition to 3: 3
Repartition to 20: 20

When is coalesce the right choice for reducing?

Coalesce is right when you are reducing partitions and there is no heavy computation upstream. The classic case is writing output after a light transformation. You avoid the shuffle cost and still get a sensible file count.

PYTHON

# After filtering, many partitions are nearly empty
txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
alerts = txn.filter(F.col("amount") > 1000000)  # very selective filter
print("Alert partitions:", alerts.rdd.getNumPartitions())  # 200, mostly empty

# Coalesce to 4: cheap, no shuffle, writing output
alerts.coalesce(4).write.mode("overwrite").parquet(
    "hdfs:///bank/high-value-alerts/2026-01/"
)
print("Written 4 files")

TEXT

Alert partitions: 200
Written 4 files

When is repartition the right choice for increasing?

Repartition is right when you need to increase parallelism or when you need balanced output. The shuffle is an unavoidable cost for creating more partitions.

PYTHON

# Small file with low parallelism: need to increase
small = spark.read.csv("hdfs:///bank/reference/branch-list.csv")
print("CSV partitions:", small.rdd.getNumPartitions())  # 1

# Repartition to increase parallelism for a heavy join
expanded = small.repartition(16)
print("After repartition:", expanded.rdd.getNumPartitions())  # 16

TEXT

CSV partitions: 1
After repartition: 16

What is the repartition(1) vs coalesce(1) trade-off?

Both end with a single partition holding all the data, but the path differs completely.

coalesce(1): narrow, no shuffle boundary. The single final task runs every upstream transformation in the same chain. All preceding work is squeezed into one task with no parallelism at all.

repartition(1): wide, shuffle boundary. The upstream transformations run with full parallelism across many tasks. Each task writes its shuffle output, and then a single reduce task gathers everything.

PYTHON

# coalesce(1): DANGEROUS for heavy upstream work
# The filter, enrichment, and aggregation ALL run in 1 task
bad = spark.read.parquet("hdfs:///bank/transactions/2026/") \
    .filter(F.col("status") == "COMPLETED") \
    .withColumn("tax", F.col("amount") * 0.18) \
    .groupBy("branch").agg(F.sum("amount").alias("total")) \
    .coalesce(1)

# repartition(1): upstream runs in parallel
# Only the final gather is single-threaded
good = spark.read.parquet("hdfs:///bank/transactions/2026/") \
    .filter(F.col("status") == "COMPLETED") \
    .withColumn("tax", F.col("amount") * 0.18) \
    .groupBy("branch").agg(F.sum("amount").alias("total")) \
    .repartition(1)

good.write.csv("hdfs:///bank/reports/annual-branch-totals.csv")

How do the physical plans differ?

The physical plan reveals the difference clearly. Coalesce shows no Exchange node; repartition shows an Exchange that acts as a stage boundary.

PYTHON

df = spark.range(1000)

print("=== coalesce(1) ===")
df.withColumn("doubled", F.col("id") * 2).coalesce(1).explain()

print("\n=== repartition(1) ===")
df.withColumn("doubled", F.col("id") * 2).repartition(1).explain()

TEXT

=== coalesce(1) ===
Coalesce 1
+- Project [id, (id * 2) AS doubled]
   +- Range (0, 1000, step=1, splits=8)

=== repartition(1) ===
Exchange RoundRobinPartitioning(1)
+- Project [id, (id * 2) AS doubled]
   +- Range (0, 1000, step=1, splits=8)

What are the risks of a single partition?

Neither single-partition option is good for large data. One partition means one task holds everything, risking out-of-memory. For writing a single file over large data, the better pattern is to keep parallelism and let the file system or a downstream step merge, or to accept multiple files.

PYTHON

# BAD: single partition on large data risks OOM
# large_df.coalesce(1).write.parquet(...)   # OOM risk
# large_df.repartition(1).write.parquet(...)  # still OOM risk

# BETTER: write multiple files, merge downstream if needed
large_df = spark.read.parquet("hdfs:///bank/transactions/2026/")
large_df.coalesce(10).write.parquet("hdfs:///bank/output/")  # 10 manageable files

# Or use maxRecordsPerFile to control file sizes
large_df.write.option("maxRecordsPerFile", 1000000).parquet(
    "hdfs:///bank/output/"
)

What are common mistakes?

The most common mistake is using coalesce(1) before heavy computation, which serializes the entire pipeline into a single task.

PYTHON

# WRONG: coalesce(1) before groupBy serializes everything
bad = txn.coalesce(1).groupBy("branch").agg(F.sum("amount"))

# CORRECT: groupBy first (parallel), then coalesce for output
good = txn.groupBy("branch").agg(F.sum("amount")).coalesce(1)

# WRONG: using coalesce(1) on large data
# Entire 10TB dataset in one task → OOM

# CORRECT: use reasonable partition count
txn.coalesce(20).write.parquet("/bank/output/")

What should a senior engineer know?

A senior engineer articulates the decision as: coalesce saves the shuffle cost and is correct when the data is already small or the upstream work is trivial. Repartition pays for a shuffle but preserves upstream parallelism and guarantees balanced output. For single-partition writes on heavy data, repartition(1) is the lesser evil because the upstream stays parallel, but the real answer is to avoid single-partition writes on large data entirely.

PYTHON

# Senior decision matrix:
# Need to REDUCE, light upstream → coalesce(n)
# Need to REDUCE, heavy upstream → repartition(n) (shuffle boundary)
# Need to INCREASE               → repartition(n) (only option)
# Need single file, small data   → coalesce(1)
# Need single file, large data   → repartition(1), or better: avoid it

Visual Overview

DIAGRAM

coalesce(1) vs repartition(1)

coalesce(1) – NO shuffle boundary:
  ┌──────────────────────────────────────────────┐
  │           ALL work in 1 TASK                  │
  │  Read → Filter → Enrich → Aggregate → Write  │
  │  (no parallelism, entire pipeline serialized) │
  └──────────────────────────────────────────────┘
  Risk: OOM on large data, very slow

repartition(1) – WITH shuffle boundary:
  Stage 1 (parallel, many tasks)     Stage 2 (1 task)
  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐    ┌──────────────┐
  │Read │ │Read │ │Read │ │Read │    │  Gather      │
  │Filt.│ │Filt.│ │Filt.│ │Filt.│    │  + Write     │
  │Enr. │ │Enr. │ │Enr. │ │Enr. │    │  (1 partition)│
  │Agg  │ │Agg  │ │Agg  │ │Agg  │    └──────────────┘
  └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘          ▲
     └───────┴───────┴───────┴──── shuffle ─┘
  Upstream is parallel. Only the gather is single-threaded.

Key Takeaway

The recommendation to coalesce when reducing and repartition when increasing comes straight from the mechanics. Coalesce is a narrow transformation that merges existing partitions on the same executor without any network movement, so it is cheap but can only reduce the count. Repartition is a wide transformation that performs a full shuffle, so it can both increase and decrease and it always produces balanced partitions. Therefore, for a plain reduction where perfect balance does not matter, coalesce saves the shuffle; to increase parallelism or force even sizes, repartition is the only option.

The clearest illustration is one partition. Both coalesce of one and repartition of one end with all data in a single partition, but the path is completely different. Coalesce of one is narrow, so there is no shuffle boundary, and the single final task is forced to also run every upstream transformation. All the heavy work is squeezed into one task with no parallelism. Repartition of one inserts a shuffle, so the upstream work runs in parallel across many tasks and only the final gather into one partition is single-threaded.

In my work at the National Bank of India I saw a regulatory report that had to be a single CSV written with coalesce of one, and because the aggregation upstream was heavy, the whole job was serialized into one task and ran for hours. Switching to repartition of one kept the aggregation parallel across two hundred tasks and only gathered at the end, and the runtime dropped dramatically. Conversely, when I only needed to shrink many tiny partitions after a light filter, coalesce was correct because there was no heavy upstream work to serialize.

So my rule in interviews is: reduce with coalesce, increase with repartition, and when collapsing to a single partition over heavy work, prefer repartition of one because coalesce of one drags the entire upstream chain into one task.

Remember one sentence: coalesce reduces cheaply with no shuffle, repartition increases or balances with a shuffle, and for a single partition over heavy work always prefer repartition to preserve upstream parallelism.

---