Definition
- After heavy filter – the DataFrame keeps the same partition count but most partitions are nearly empty. Repartition rebalances into fewer, evenly-sized partitions to restore efficient parallelism.
- Before writing – each partition becomes one output file. Coalesce reduces the partition count cheaply so the job does not emit hundreds of tiny files that hurt downstream reads.
- Key distinction – repartition fixes skewed parallelism (worth a shuffle); coalesce fixes the small-file problem (no shuffle needed).
Suppose the National Bank of India starts with two hundred clerks each holding a stack of transaction slips. A filter keeping only fraud alerts leaves ninety-nine percent of clerks with empty hands and one clerk buried under slips. Repartition re-deals so every clerk has a fair share again. Later, when filing the final report, writing one tiny file per clerk floods the records room with two hundred almost-empty folders, so you coalesce the stacks into ten tidy folders before filing.
Understanding the Concepts
Why does a filter leave many empty partitions?
A filter in Spark is a narrow transformation. It does not change the partition count; it only drops rows inside each existing partition. So if you start with two hundred partitions and keep only one percent of rows, you still have two hundred partitions, but most are nearly empty and a couple may hold all the survivors.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.appName("NBI-Filter-Repart").getOrCreate()
# 200 partitions of transactions
txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
print("Before filter:", txn.rdd.getNumPartitions()) # 200
# Highly selective filter: only fraud alerts
alerts = txn.filter(F.col("fraud_flag") == True)
print("After filter:", alerts.rdd.getNumPartitions()) # still 200
# Most partitions are empty
sizes = alerts.rdd.mapPartitions(lambda it: [sum(1 for _ in it)]).collect()
empty = sum(1 for s in sizes if s == 0)
print(f"Empty partitions: {empty} out of {len(sizes)}")
Before filter: 200
After filter: 200
Empty partitions: 195 out of 200
When should you repartition after a heavy filter?
Repartition after a filter when the downstream work is heavy and needs balanced parallelism. The survivors are clustered in a few partitions, so Spark schedules two hundred tasks but most do nothing while a few are overloaded. Repartitioning to a smaller, sensible number rebalances the survivors.
# BAD: 200 tasks, 195 empty, 5 overloaded
scored = alerts.withColumn(
"risk_score",
F.when(F.col("amount") > 500000, 0.95).otherwise(0.50)
)
# Only 5 tasks do real work, rest are wasted
# GOOD: repartition to rebalance before heavy work
alerts_balanced = alerts.repartition(8)
scored = alerts_balanced.withColumn(
"risk_score",
F.when(F.col("amount") > 500000, 0.95).otherwise(0.50)
)
# 8 tasks, all roughly equal work
print("Balanced partitions:", scored.rdd.getNumPartitions())
Balanced partitions: 8
When should you coalesce before writing?
Coalesce before writing when you need to reduce the file count. Each partition becomes one output file. If you write two hundred partitions of tiny data, you emit two hundred tiny files, which is a well-known anti-pattern that hurts downstream readers.
# BAD: 200 tiny files
alerts.write.mode("overwrite").parquet("/bank/fraud-alerts/bad/")
# Creates 200 files, 195 of which are empty or near-empty
# GOOD: coalesce to a reasonable file count
alerts.coalesce(4).write.mode("overwrite").parquet(
"/bank/fraud-alerts/good/"
)
# Creates 4 files, reasonable size
print("Written 4 output files")
Written 4 output files
How do these two scenarios differ?
They solve different problems. Repartition fixes skewed parallelism and is worth the shuffle cost. Coalesce fixes the small-file problem and avoids paying for a shuffle.
| Scenario | Problem | Solution | Why |
|---|---|---|---|
| After heavy filter, before heavy work | Skewed parallelism | repartition(n) | Need balanced tasks |
| Before writing output | Too many tiny files | coalesce(n) | Need fewer files, no heavy work |
| After heavy filter, directly writing | Both problems | coalesce(n) or repartition(n) | Depends on upstream work |
# Full pipeline: filter → repartition → heavy work → coalesce → write
txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
alerts = txn.filter(F.col("fraud_flag") == True)
alerts_balanced = alerts.repartition(8) # fix parallelism
scored = alerts_balanced.withColumn(
"risk_score",
F.when(F.col("amount") > 500000, 0.95).otherwise(0.50)
)
# Coalesce to 2 files for output
scored.coalesce(2).write.mode("overwrite").parquet(
"hdfs:///bank/fraud-scored/2026-01/"
)
What is the small-file problem?
Tiny files are a well-known anti-pattern in data engineering. Object stores and HDFS pay a fixed cost per file. Listing and opening many files is slow. Downstream readers lose efficiency because small files defeat columnar read optimizations like predicate pushdown and vectorized reads.
# Demonstrating the small-file problem
import os
# BAD: 200 tiny files in a directory
alerts.write.mode("overwrite").parquet("/tmp/tiny-files/")
files = [f for f in os.listdir("/tmp/tiny-files/") if f.endswith(".parquet")]
print(f"File count: {len(files)}")
print(f"Average file size: {sum(os.path.getsize(f'/tmp/tiny-files/{f}') for f in files) / len(files):.0f} bytes")
# GOOD: 4 well-sized files
alerts.coalesce(4).write.mode("overwrite").parquet("/tmp/good-files/")
files = [f for f in os.listdir("/tmp/good-files/") if f.endswith(".parquet")]
print(f"File count: {len(files)}")
File count: 200
Average file size: 1245 bytes
File count: 4
What if there is heavy computation right before the write?
If there is heavy computation immediately before the write, coalescing to a tiny number can serialize that work. In that case, use repartition instead to preserve upstream parallelism.
# WRONG: coalesce(1) before heavy aggregation serializes everything
bad = txn.filter(F.col("status") == "COMPLETED") \
.groupBy("branch", "account_id") \
.agg(F.sum("amount"), F.count("*")) \
.coalesce(1) # entire aggregation in 1 task
# CORRECT: do heavy work first, then coalesce
good = txn.filter(F.col("status") == "COMPLETED") \
.groupBy("branch", "account_id") \
.agg(F.sum("amount"), F.count("*"))
# After aggregation, result is small, safe to coalesce
good.coalesce(1).write.csv("/bank/reports/summary.csv")
# ALSO CORRECT: repartition for heavy upstream
better = txn.filter(F.col("status") == "COMPLETED") \
.groupBy("branch", "account_id") \
.agg(F.sum("amount"), F.count("*")) \
.repartition(1) # shuffle boundary keeps aggregation parallel
What are common mistakes?
The most common mistakes are writing without reducing partitions (tiny files) and repartitioning when coalesce would suffice (paying for an unnecessary shuffle).
# WRONG: writing 200 tiny files after a filter
alerts.write.parquet("/bank/output/") # 200 files, most empty
# WRONG: repartition before a simple write (unnecessary shuffle)
alerts.repartition(4).write.parquet("/bank/output/") # works but pays shuffle cost
# CORRECT: coalesce before a simple write (no shuffle needed)
alerts.coalesce(4).write.parquet("/bank/output/") # 4 files, no shuffle
# WRONG: coalesce to 1 before heavy work
alerts.coalesce(1).withColumn("x", F.col("amount") * 2).write.parquet("/bank/output/")
# CORRECT: heavy work first, then coalesce
alerts.withColumn("x", F.col("amount") * 2).coalesce(1).write.parquet("/bank/output/")
What should a senior engineer know?
A senior engineer distinguishes the two problems and applies the matching tool. They also know that AQE (Adaptive Query Execution) in Spark 3 can automatically coalesce shuffle partitions when they are too small, reducing the need for manual coalesce in some cases, but it only applies after shuffles, not after narrow transformations like filters.
# AQE automatic coalesce of shuffle partitions
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
# After a shuffle (groupBy), AQE may automatically reduce tiny partitions
result = txn.groupBy("branch").agg(F.sum("amount"))
result.write.parquet("/bank/output/") # AQE may merge small shuffle partitions
# But after a filter (no shuffle), AQE does not help
# You still need manual coalesce or repartition
alerts = txn.filter(F.col("fraud_flag") == True)
alerts.coalesce(4).write.parquet("/bank/alerts/") # manual coalesce needed
Visual Overview
After Heavy Filter: Before repartition After repartition(4) ┌───┐ ┌───┐ ┌───┐ ┌───┐ ┌──────┐ ┌──────┐ │ 0 │ │ 0 │ │500│ │ 0 │ │ 125 │ │ 125 │ └───┘ └───┘ └───┘ └───┘ └──────┘ └──────┘ ┌───┐ ┌───┐ ┌───┐ ┌───┐ ┌──────┐ ┌──────┐ │ 0 │ │ 0 │ │ 0 │ │ 0 │ │ 125 │ │ 125 │ └───┘ └───┘ └───┘ └───┘ └──────┘ └──────┘ 200 partitions, 195 empty 4 partitions, balanced (wasteful tasks) (efficient tasks) Before Writing: Without coalesce With coalesce(4) ┌─┐┌─┐┌─┐┌─┐┌─┐ ... ┌─┐ ┌────┐┌────┐┌────┐┌────┐ │ ││ ││ ││ ││ │ │ │ │file││file││file││file│ └─┘└─┘└─┘└─┘└─┘ ... └─┘ │ 1 ││ 2 ││ 3 ││ 4 │ 200 tiny files └────┘└────┘└────┘└────┘ (small-file problem) 4 reasonable files
Key Takeaway
These are two different problems that happen to both involve partition counts. A filter in Spark is a narrow transformation, so it never changes the number of partitions: it only drops rows inside each one. That means a highly selective filter leaves you with the same large number of partitions, but most are nearly empty while a few hold the survivors. That layout is wasteful because Spark still schedules a task per partition and any downstream wide operation shuffles all of them. So after a heavy filter I repartition to a small, sensible number, accepting the shuffle cost because it rebalances the survivors into evenly sized partitions and restores efficient parallelism.
The second problem is at write time. Each partition becomes one output file, so if I write without reducing the count I emit a flood of tiny files, which object stores and HDFS handle poorly: fixed cost per file, slow listings, and defeated columnar read optimizations downstream. So before writing I coalesce to a reasonable file count. Coalesce is the right tool here because it is a cheap narrow merge and there is usually no heavy computation between the merge and the write, so it does not collapse any important parallelism.
In my fraud pipeline at the National Bank of India this was the exact flow. The fraud filter left two hundred partitions almost all empty with the few survivors clustered in two or three partitions, so I repartitioned to four to rebalance before the enrichment join, which turned two hundred mostly-idle tasks into four efficient ones. Then, because the alert volume was small, I coalesced to one or two partitions before writing so the data lake received a couple of tidy files instead of two hundred tiny ones.
The way I summarize it in interviews is: repartition after a selective filter to fix skewed parallelism, and coalesce before writing to fix the small-file problem. And if there is heavy work right before the write, repartition instead so coalesce does not serialize it.
Remember one sentence: repartition after a heavy filter rebalances skewed partitions, and coalesce before writing avoids a flood of tiny files.
---