Back to Apache Spark: Core
Q25Question 25 of 72

How Spark handles schema inference

Key Concepts
  • CSV inference with inferSchema=true reads the data twice: once to deduce types, once to load; without it, all columns default to string.
  • JSON inference inspects a configurable sample of records and merges fields into a superset schema, which can miss fields or infer types too narrowly.
  • Parquet and ORC store the complete schema in the file footer, so no inference is needed and types are exact.
  • An explicit schema in production eliminates the double-read for CSV, removes sampling risk for JSON, and prevents schema drift.
  • An explicit schema also acts as a data contract: unexpected source changes surface as visible failures at read time.
  • Pair an explicit schema with failfast read mode and a corrupt-record column for production-grade error handling.

Definition

CSV inference reads the data twice, JSON infers from a sample, Parquet/ORC carry the schema in the footer so no inference is needed, and an explicit schema in production avoids the double read and prevents schema drift.
  • CSV – inferSchema=true triggers a double read: Spark scans once to deduce types, then loads; without it, all columns are string.
  • JSON – Spark inspects a configurable sample (samplingRatio) and merges fields/types into a superset schema.
  • Parquet/ORC – schema is embedded in the file footer at write time; Spark reads it directly, no inference.
  • Explicit schema – providing a StructType with schema() eliminates the extra pass, removes sampling risk, and prevents schema drift.

At the National Bank of India imagine receiving boxes of transaction forms with no labels on the columns. For CSV, the clerk reads the whole box once just to figure out the columns, then reads it again to file them: twice the work. For JSON, the clerk reads a sample stack and guesses. For Parquet and ORC, the labels are printed on the cover, so no guessing is needed. In production you hand the clerk a printed column key up front (explicit schema), so there is no double reading and no risk of guessing wrong.

Understanding the Concepts

What is schema inference in Spark?

Schema inference is Spark's process of deducing column names and data types when they are not explicitly declared. It behaves differently for each file format.

PYTHON
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, DateType

spark = SparkSession.builder.appName("NBI-Schema-Inference").getOrCreate()

How does CSV inference work?

For CSV, Spark cannot know types from the raw text. With inferSchema=true, Spark launches a job that scans the data to determine each column's type, then a second job reads the data using that schema. This double read is expensive on large files.

PYTHON
# CSV without inferSchema: all columns are strings
df_string = spark.read.csv("hdfs:///bank/transactions/daily.csv", header=True)
df_string.printSchema()

# CSV with inferSchema: double read, types deduced
df_inferred = spark.read.csv(
    "hdfs:///bank/transactions/daily.csv",
    header=True,
    inferSchema=True
)
df_inferred.printSchema()
DIAGRAM
# Without inferSchema:
root
 |-- account_id: string
 |-- amount: string      ← everything is string
 |-- txn_date: string

# With inferSchema:
root
 |-- account_id: string
 |-- amount: double      ← inferred from data
 |-- txn_date: string    ← date not always detected

How does JSON inference work?

For JSON, Spark infers the schema by sampling records and merging fields and types. Different records can have different fields, and Spark unions them into a superset. Fields outside the sample can be missed.

PYTHON
# JSON inference: inspects a sample of records
df_json = spark.read.json("hdfs:///bank/api-events/2026-01-15/")
df_json.printSchema()

# Control sampling ratio (default 1.0 = all records)
df_sampled = spark.read.option("samplingRatio", "0.1").json(
    "hdfs:///bank/api-events/2026-01-15/"
)
# Risk: field that appears only outside the 10% sample is missed
TEXT
root
 |-- account_id: string
 |-- amount: double
 |-- event_type: string
 |-- metadata: struct
 |    |-- source: string
 |    |-- timestamp: string

How does Parquet/ORC handle schemas?

Parquet and ORC are columnar formats that write the complete schema in the file footer at write time. When Spark reads them, it loads the schema directly from the footer. No inference, no double read, and the types are exactly what was written.

PYTHON
# Parquet: schema embedded in file footer
df_parquet = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
df_parquet.printSchema()
# Schema is exact: no inference, no guessing
DIAGRAM
root
 |-- account_id: string
 |-- amount: decimal(18,2)   ← exact type from writer
 |-- txn_date: date
 |-- branch: string

Why should you provide an explicit schema in production?

An explicit schema eliminates the double read for CSV, removes sampling risk for JSON, and prevents schema drift where inferred types change between runs.

PYTHON
# Define explicit schema
txn_schema = StructType([
    StructField("account_id", StringType(), False),
    StructField("amount", DoubleType(), False),
    StructField("txn_date", DateType(), True),
    StructField("branch", StringType(), True)
])

# Read CSV with explicit schema: single pass, no inference
df = spark.read.schema(txn_schema).csv(
    "hdfs:///bank/transactions/daily.csv",
    header=True
)
df.printSchema()
df.show(3)
TEXT
root
 |-- account_id: string (nullable = false)
 |-- amount: double (nullable = false)
 |-- txn_date: date (nullable = true)
 |-- branch: string (nullable = true)

+----------+--------+----------+------+
|account_id|  amount|  txn_date|branch|
+----------+--------+----------+------+
|   SAV2001| 50000.0|2026-01-15|Mumbai|
|   SAV2002| 12000.0|2026-01-15| Delhi|
|   CUR3001|300000.0|2026-01-16|Chennai|
+----------+--------+----------+------+

How does an explicit schema act as a data contract?

By declaring the expected schema, you turn an unexpected change in the source into a visible failure at read time. This is far safer than silently coercing a monetary field to the wrong type.

PYTHON
# If the source adds a new column or changes a type:
# With explicit schema + failfast mode, the job fails immediately
df_strict = spark.read \
    .schema(txn_schema) \
    .option("mode", "FAILFAST") \
    .csv("hdfs:///bank/transactions/daily.csv", header=True)

# If a row has a non-numeric value in the amount column:
# FAILFAST mode throws an exception immediately
# PERMISSIVE mode (default) puts the row in a _corrupt_record column
# DROPMALFORMED mode silently drops the row

# Production pattern: PERMISSIVE with corrupt record column
df_safe = spark.read \
    .schema(txn_schema.add("_corrupt_record", StringType())) \
    .option("mode", "PERMISSIVE") \
    .option("columnNameOfCorruptRecord", "_corrupt_record") \
    .csv("hdfs:///bank/transactions/daily.csv", header=True)

# Check for bad rows
bad_rows = df_safe.filter(df_safe._corrupt_record.isNotNull())
print(f"Corrupt rows: {bad_rows.count()}")
TEXT
Corrupt rows: 3

How do you prevent schema drift?

Schema drift is when inferred types silently change between runs as new data arrives. An explicit schema prevents this by enforcing the same types every time.

PYTHON
# Schema drift example:
# Day 1: amount column has values like "50000" → inferred as integer
# Day 2: amount column has values like "50000.50" → inferred as double
# Downstream jobs expecting integer break or produce wrong results

# Fix: explicit schema enforces double every time
txn_schema = StructType([
    StructField("account_id", StringType(), False),
    StructField("amount", DoubleType(), False),  # always double
    StructField("txn_date", DateType(), True),
    StructField("branch", StringType(), True)
])

# Now Day 1 and Day 2 both read amount as double: consistent

How do the formats compare?

| Feature | CSV | JSON | Parquet/ORC |
|---|---|---|---|
| Schema source | Inferred or explicit | Inferred or explicit | Embedded in footer |
| Inference cost | Double read | Sample scan | None |
| Type accuracy | Can be wrong | Can be narrow | Exact |
| Production recommendation | Always explicit | Always explicit | Schema travels with data |

PYTHON
# Best practice summary:
# CSV:     spark.read.schema(my_schema).csv(path)
# JSON:    spark.read.schema(my_schema).json(path)
# Parquet: spark.read.parquet(path)  # schema embedded, but can override
# ORC:     spark.read.orc(path)      # schema embedded

What should a senior engineer know?

A senior engineer provides explicit schemas everywhere in production, reserves inference for one-off exploration, and prefers Parquet or ORC so the schema travels with the data. They pair explicit schemas with failfast or permissive read modes for production-grade error handling.

PYTHON
# Senior production pattern:
# 1. Define schema in code (or read from a schema registry)
# 2. Read with explicit schema and PERMISSIVE mode
# 3. Check corrupt records
# 4. Write as Parquet so downstream gets embedded schema

txn = spark.read \
    .schema(txn_schema) \
    .option("mode", "PERMISSIVE") \
    .csv("hdfs:///bank/incoming/daily.csv", header=True)

# Validate: no corrupt records
assert txn.filter("_corrupt_record IS NOT NULL").count() == 0, "Bad rows found!"

# Write as Parquet: schema embedded in footer
txn.write.parquet("hdfs:///bank/curated/transactions/2026-01-15/")

Visual Overview

DIAGRAM
Schema Inference by Format:

CSV (inferSchema=true):
  Pass 1: Scan → deduce types (integer? double? string?)
  Pass 2: Read → load data with inferred schema
  Cost: 2x I/O. Risk: wrong types from sample.

JSON (samplingRatio):
  Sample N% → merge fields → superset schema
  Cost: sample scan. Risk: missed fields, narrow types.

Parquet / ORC:
  ┌──────────────────────────────────┐
  │  File Footer                     │
  │  Schema: account_id STRING,      │
  │          amount DECIMAL(18,2),   │
  │          txn_date DATE           │
  └──────────────────────────────────┘
  Cost: none. Types: exact.

Explicit Schema (production):
  StructType([...])  →  spark.read.schema(my_schema).csv(path)
  ├── Eliminates double read (CSV)
  ├── Removes sampling risk (JSON)
  ├── Prevents schema drift
  └── Acts as a data contract (fail on mismatch)

Key Takeaway

Schema inference is how Spark figures out column names and types when they are not declared, and it behaves differently per format. For CSV, where everything is text, inference with inferSchema=true makes Spark read the data twice: one pass to deduce the types and a second to load. For JSON, Spark infers from a configurable sample, merging fields into a superset, which can miss fields or infer types too narrowly. For Parquet and ORC there is no inference at all, because the complete schema is stored in the file footer.

In production I always provide an explicit schema. Constructing a StructType and passing it with the schema method eliminates the double read for CSV, removes the sampling risk for JSON, and prevents schema drift where inferred types silently change between runs. Just as important, an explicit schema acts as a data contract: a new column, a type change, or a renamed field in the source becomes a visible failure at read time, which in a bank is far safer than silently coercing a monetary field to the wrong type.

At the National Bank of India our daily transaction extracts arrived as large CSV files, and switching from inferSchema to explicit schemas halved the read cost by removing the double pass, while also catching a real incident where the source added a column and our strict schema failed the job visibly instead of mis-mapping the data. We wrote the curated data as Parquet so the authoritative schema traveled with it.

The practice I recommend in interviews is explicit schemas everywhere in production, inference only for one-off exploration, and Parquet or ORC as the storage format so the schema is embedded in the footer.

Remember one sentence: provide an explicit schema in production to avoid the double read, prevent schema drift, and turn unexpected source changes into visible failures.

---

Share
Use ← → arrow keys to navigate