Definition
- explode() – flattens an array or map column into multiple rows, one per element, dropping rows where the array is null or empty.
- explode_outer() – same flattening, but keeps rows with null or empty arrays by emitting a single row with null in the exploded column.
- Row multiplication – exploding is a one-to-many operation; every parent column is copied onto every new row, so filtering and selecting before the explode keeps the intermediate small.
- posexplode – variants that additionally emit the element's position index.
Each National Bank customer has a list of recent transactions. Exploding turns one customer row into one row per transaction. With explode, a customer who made no transactions vanishes from the result, which can silently drop customers from a report. With explode_outer, that customer still appears once with a null transaction. And because exploding copies the customer's other columns onto every transaction row, you trim to only the columns you need first, or you waste memory copying everything.
Understanding the Concepts
What does explode() do?
explode() takes an array column and produces one row per element, copying the other columns of the parent row onto each new row. If the array is null or empty, the parent row is dropped.
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.types import ArrayType, StringType, StructType, StructField, LongType
spark = SparkSession.builder.appName("ExplodeDemo").getOrCreate()
data = [
(1001, "Rajesh", [50000, 30000, 10000]),
(1002, "Priya", [80000]),
(1003, "Pankaj", []), # empty array
(1004, "Meera", None), # null array
]
df = spark.createDataFrame(data, ["account_id", "holder", "txn_amounts"])
result = df.select("account_id", "holder", F.explode("txn_amounts").alias("amount"))
result.show()
+----------+------+------+
|account_id|holder|amount|
+----------+------+------+
| 1001|Rajesh| 50000|
| 1001|Rajesh| 30000|
| 1001|Rajesh| 10000|
| 1002| Priya| 80000|
+----------+------+------+
Pankaj (empty array) and Meera (null) are gone. This behaves like an inner join between the row and its array elements.
What does explode_outer() do?
explode_outer() is the left-outer equivalent. Rows with null or empty arrays still emit one row with null in the exploded column.
result = df.select(
"account_id", "holder",
F.explode_outer("txn_amounts").alias("amount")
)
result.show()
+----------+------+------+
|account_id|holder|amount|
+----------+------+------+
| 1001|Rajesh| 50000|
| 1001|Rajesh| 30000|
| 1001|Rajesh| 10000|
| 1002| Priya| 80000|
| 1003|Pankaj| null|
| 1004| Meera| null|
+----------+------+------+
Every parent row is preserved. This is the safe default when the report must account for every customer.
Why does the choice matter for correctness?
Using explode when parent rows must be preserved silently drops data, which can cause incorrect totals in reports.
# Wrong: using explode for a customer count report
count_wrong = df.select(
"account_id", F.explode("txn_amounts")
).select("account_id").distinct().count()
print(f"Customer count (explode): {count_wrong}") # 2, not 4
# Right: using explode_outer preserves all customers
count_right = df.select(
"account_id", F.explode_outer("txn_amounts")
).select("account_id").distinct().count()
print(f"Customer count (explode_outer): {count_right}") # 4
Why should you filter and select BEFORE exploding?
Explode is a one-to-many operation. A parent row with ten elements becomes ten rows, and every column you carry through is copied ten times.
# Bad: explode first, then filter and select
# All 100 columns are duplicated per element, then most are discarded
bad = df.withColumn("amt", F.explode("txn_amounts")) \
.filter(F.col("amt") > 20000) \
.select("account_id", "amt")
# Good: filter and select first, then explode
# Only 2 columns are duplicated per element
good = df.select("account_id", "txn_amounts") \
.filter(F.size("txn_amounts") > 0) \
.withColumn("amt", F.explode("txn_amounts")) \
.filter(F.col("amt") > 20000)
The "good" version builds a much smaller intermediate because only the needed columns are carried through the multiplication.
What are posexplode and posexplode_outer?
These variants additionally emit the element's zero-based position index.
result = df.select(
"account_id",
F.posexplode_outer("txn_amounts").alias("pos", "amount")
)
result.show()
+----------+----+------+
|account_id| pos|amount|
+----------+----+------+
| 1001| 0| 50000|
| 1001| 1| 30000|
| 1001| 2| 10000|
| 1002| 0| 80000|
| 1003|null| null|
| 1004|null| null|
+----------+----+------+
The position is useful when the order of elements carries meaning, such as the sequence of transactions.
How do you explode a map column?
Exploding a map produces key and value columns.
map_data = [
(1001, {"savings": 50000, "current": 120000}),
(1002, {"savings": 30000}),
(1003, None),
]
map_df = spark.createDataFrame(map_data, ["account_id", "balances"])
result = map_df.select(
"account_id",
F.explode_outer("balances").alias("account_type", "balance")
)
result.show()
+----------+------------+-------+
|account_id|account_type|balance|
+----------+------------+-------+
| 1001| savings| 50000|
| 1001| current| 120000|
| 1002| savings| 30000|
| 1003| null| null|
+----------+------------+-------+
What happens with multiple arrays?
Exploding two arrays separately creates a cross-product. Use arrays_zip to pair elements instead.
data = [(1001, [50000, 30000], ["ATM", "NEFT"])]
df2 = spark.createDataFrame(data, ["account_id", "amounts", "channels"])
# Wrong: cross-product (2 x 2 = 4 rows)
cross = df2.withColumn("amt", F.explode("amounts")) \
.withColumn("ch", F.explode("channels"))
print(f"Cross product rows: {cross.count()}") # 4
# Right: arrays_zip pairs by index (2 rows)
zipped = df2.withColumn(
"paired", F.explode(F.arrays_zip("amounts", "channels"))
).select("account_id", "paired.amounts", "paired.channels")
print(f"Zipped rows: {zipped.count()}") # 2
What should a senior engineer know?
- Default to
explode_outerwhen parent rows must be preserved; useexplodeonly when dropping empties is intentional. - Always select and filter before exploding to minimize the multiplied intermediate.
- For element-wise transforms without needing one-row-per-element output, prefer higher-order array functions (
transform,filter,aggregate) which avoid the multiplication entirely. - Watch for accidental cross-products when exploding multiple arrays.
# Higher-order alternative: no row multiplication
result = df.withColumn(
"large_txns",
F.filter("txn_amounts", lambda x: x > 20000)
)
What are common interview follow-up questions?
- What is the difference between explode and lateral view?
LATERAL VIEWis the SQL syntax equivalent ofexplodein aSELECTstatement. - Can you explode a struct? No, explode works on arrays and maps. For structs, use
col.field_nameto access fields. - Does explode trigger a shuffle? No, explode is a narrow transformation that operates within each partition.
Visual Overview
explode() vs explode_outer()
────────────────────────────
Input:
┌──────────┬─────────────────┐
│ Rajesh │ [50K, 30K, 10K] │
│ Priya │ [80K] │
│ Pankaj │ [] │
│ Meera │ null │
└──────────┴─────────────────┘
explode(): explode_outer():
┌──────────┬───────┐ ┌──────────┬───────┐
│ Rajesh │ 50K │ │ Rajesh │ 50K │
│ Rajesh │ 30K │ │ Rajesh │ 30K │
│ Rajesh │ 10K │ │ Rajesh │ 10K │
│ Priya │ 80K │ │ Priya │ 80K │
└──────────┴───────┘ │ Pankaj │ null │
Pankaj and Meera DROPPED │ Meera │ null │
└──────────┴───────┘
ALL rows PRESERVED
Key Takeaway
Both explode and explode_outer flatten an array or map column into multiple rows, one per element, copying the parent row's other columns onto each new row. The difference is null handling. explode behaves like an inner join: a parent whose array is null or empty contributes zero rows and disappears. explode_outer behaves like a left-outer join: that parent still produces one row with null in the exploded column.
The choice is a correctness decision. If a report must account for every customer, using explode silently drops customers with no transactions, understating totals. So when parent rows must be preserved, explode_outer is the safe default, and plain explode is for when dropping empties is intentional.
The performance guidance is to select and filter before exploding, because explode is a one-to-many operation. A parent row with ten elements becomes ten rows, and every column carried through is copied ten times. If you explode first and trim later, Spark builds a wide, multiplied intermediate and then throws most of it away. At the National Bank of India, our statement job once used explode and silently dropped a new customer with zero transactions, which auditors caught. We switched to explode_outer and moved column selection ahead of the explode so heavy KYC fields were never copied onto every transaction line.
Remember one sentence: explode drops null and empty arrays, explode_outer keeps them as nulls, and always trim and filter before you explode to avoid multiplying unnecessary data.
---