Definition
- void / NullType – the type Spark assigns to a literal null when no other type can be inferred; most commonly from
lit(None)without a cast. - Harmless in memory – shows as null in
show()and DataFrame operations. - Fails at write – Parquet, ORC, and Delta require a concrete physical type per column; void has none, so the writer throws an error.
- Fix – cast the null to the intended type:
lit(None).cast('string'),lit(None).cast('double'), etc.
Imagine adding a blank column to a National Bank report but never saying what kind of data it should hold: text, number, or date. A screen display tolerates the mystery blank, but when you try to file it into the bank's strict typed archive (Parquet or Delta), the archivist refuses because every column must declare its type.
Understanding the Concepts
What is the void datatype?
When you write F.lit(None), Spark has no information about what type the null should be, so it infers NullType, displayed as void in the schema. This also appears when a column is derived entirely from nulls, or from a CASE expression where every branch is null.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.appName("NBI-Void-Type").getOrCreate()
df = spark.createDataFrame([
("SAV2001", 50000.00),
("SAV2002", 12000.00)
], ["account_id", "amount"])
# Adding a column with lit(None): creates void type
df_with_void = df.withColumn("middle_name", F.lit(None))
df_with_void.printSchema()
df_with_void.show()
root |-- account_id: string |-- amount: double |-- middle_name: void ← NullType! +----------+--------+-----------+ |account_id| amount|middle_name| +----------+--------+-----------+ | SAV2001| 50000.0| null| | SAV2002| 12000.0| null| +----------+--------+-----------+
Why does void cause errors when writing?
Parquet, ORC, and Delta are columnar formats that store a physical type per column in their metadata. A void column has no physical type, so the writer cannot encode it and throws an error.
# This FAILS:
try:
df_with_void.write.mode("overwrite").parquet("/tmp/void-test/")
except Exception as e:
print(f"Error: {e}")
# Delta also rejects void:
try:
df_with_void.write.format("delta").mode("overwrite").save("/tmp/void-delta/")
except Exception as e:
print(f"Delta error: {e}")
Error: Parquet data source does not support void data type.
Delta error: NullType columns are not supported in Delta tables.
How do you fix the void datatype?
Cast the null literal to a concrete type. Choose the type that matches the column's intended meaning.
# Fix: cast to the intended type
df_fixed = df.withColumn("middle_name", F.lit(None).cast("string"))
df_fixed.printSchema()
df_fixed.show()
# Now writing succeeds
df_fixed.write.mode("overwrite").parquet("/tmp/void-fixed/")
print("Write succeeded!")
root |-- account_id: string |-- amount: double |-- middle_name: string ← concrete type! +----------+--------+-----------+ |account_id| amount|middle_name| +----------+--------+-----------+ | SAV2001| 50000.0| null| | SAV2002| 12000.0| null| +----------+--------+-----------+ Write succeeded!
When does void commonly appear?
The most common scenario is adding placeholder columns to align schemas for a union or merge where one source lacks a field.
# Two feeds with different schemas:
feed_a = spark.createDataFrame([
("SAV2001", 50000.00, "Rajesh Kumar")
], ["account_id", "amount", "holder_name"])
feed_b = spark.createDataFrame([
("SAV2002", 12000.00)
], ["account_id", "amount"])
# Add missing column to feed_b for union:
# WRONG: creates void type
feed_b_bad = feed_b.withColumn("holder_name", F.lit(None))
feed_b_bad.printSchema() # holder_name: void
# CORRECT: cast to string
feed_b_good = feed_b.withColumn("holder_name", F.lit(None).cast("string"))
feed_b_good.printSchema() # holder_name: string
# Union and write
result = feed_a.unionByName(feed_b_good)
result.write.mode("overwrite").parquet("/tmp/union-result/")
# Bad:
root
|-- holder_name: void
# Good:
root
|-- holder_name: string (nullable = true)
What other expressions produce void?
Besides lit(None), a CASE/WHEN expression where every branch returns null also produces void.
# CASE with all-null branches: produces void
df_case = df.withColumn(
"status",
F.when(F.col("amount") < 0, None) # all branches return None
.otherwise(None)
)
df_case.printSchema() # status: void
# Fix: cast the result
df_case_fixed = df.withColumn(
"status",
F.when(F.col("amount") < 0, F.lit(None).cast("string"))
.otherwise(F.lit(None).cast("string"))
)
df_case_fixed.printSchema() # status: string
How do you detect void columns before writing?
Always check printSchema() for any void type before writing to a typed sink.
from pyspark.sql.types import NullType
def check_for_void(df, label="DataFrame"):
void_cols = [f.name for f in df.schema.fields if isinstance(f.dataType, NullType)]
if void_cols:
print(f"WARNING: {label} has void columns: {void_cols}")
print("Fix: cast them to concrete types before writing")
else:
print(f"{label}: no void columns, safe to write")
check_for_void(df_with_void, "Bad DataFrame")
check_for_void(df_fixed, "Fixed DataFrame")
WARNING: Bad DataFrame has void columns: ['middle_name']
Fix: cast them to concrete types before writing
Fixed DataFrame: no void columns, safe to write
What are common mistakes?
The most common mistake is using lit(None) without a cast when adding placeholder columns for unions or merges.
# WRONG: lit(None) without cast
df.withColumn("settlement_date", F.lit(None)) # void!
# CORRECT: always cast
df.withColumn("settlement_date", F.lit(None).cast("date")) # date
df.withColumn("fee_amount", F.lit(None).cast("decimal(18,2)")) # decimal
df.withColumn("notes", F.lit(None).cast("string")) # string
What should a senior engineer know?
A senior engineer makes it a habit to always cast null literals to the target type when introducing placeholder columns. They add a void-check step to their pipeline before writing to any typed sink.
# Senior pattern: safe null column addition
def add_null_column(df, col_name, col_type):
"""Add a typed null column, never void."""
return df.withColumn(col_name, F.lit(None).cast(col_type))
result = add_null_column(df, "middle_name", "string")
result = add_null_column(result, "fee", "decimal(18,2)")
result = add_null_column(result, "settlement_date", "date")
check_for_void(result, "Final result")
result.write.format("delta").save("/bank/output/")
Final result: no void columns, safe to write
Visual Overview
The void (NullType) Problem:
F.lit(None) → void (NullType)
F.lit(None).cast("string") → string (nullable)
In Memory:
┌──────────┬────────┬─────────────┐
│account_id│ amount │ middle_name │
│ SAV2001 │50000.0 │ null │ ← looks fine
└──────────┴────────┴─────────────┘
At Write Time (Parquet/Delta):
┌─────────────────────────────────────┐
│ Parquet Footer needs a type: │
│ account_id: STRING ✓ │
│ amount: DOUBLE ✓ │
│ middle_name: ??? ✗ → ERROR! │
└─────────────────────────────────────┘
Fix: Cast before writing
F.lit(None).cast("string")
┌─────────────────────────────────────┐
│ Parquet Footer: │
│ account_id: STRING ✓ │
│ amount: DOUBLE ✓ │
│ middle_name: STRING ✓ (nullable) │
└─────────────────────────────────────┘
Key Takeaway
The void datatype, which Spark calls NullType, is the type Spark assigns to a literal null when it has no other information to infer a type, and the classic way to create it is lit(None) without a cast. It represents a column that is always null with no concrete type. In memory and in show() it is harmless, but the problem appears when you try to write to Parquet, ORC, or Delta, because those formats need a physical type per column and void has none.
The fix is simple: give the null a concrete type with a cast. Lit(None).cast('string') produces a nullable string column, and I choose the type that matches the column's intended meaning. If a column became void through inference rather than a literal, I cast it to the intended type before writing.
At the National Bank of India this bit us when we unioned two account feeds where one feed lacked a middle_name field. The team added that column with lit(None) to align the schemas, which produced a void column, and the write to Delta failed with a NullType error. The one-line fix was to cast the null literal to string.
The habit I adopted is to always cast null literals to the target type when introducing placeholder columns for unions or merges, and to check printSchema for any void type before writing to a typed sink.
Remember one sentence: lit(None) creates a void (NullType) column that typed formats reject; always cast it to the intended type with lit(None).cast('string') before writing.
---