Spark read modes: PERMISSIVE, DROPMALFORMED, FAILFAST | Data Engineering

Q: What are Spark read modes?

When Spark reads semistructured files like CSV or JSON, some records may not match the expected schema. The mode option controls how Spark handles these malformed records. There are three choices: tolerate, discard, or reject. from pyspark.sql import SparkSession from pyspark.sql.types import Struct

Q: What are the gotchas with _corrupt_record?

The _corrupt_record column has several practical quirks that catch people in production. GOTCHA 1: _corrupt_record must be in the schema explicitly If you omit it, PERMISSIVE still nullifies bad fields but does NOT save the raw text, so you lose visibility into what went wrong. schema_without_corrup

Q: What should a senior engineer know?

PERMISSIVE with alerting is the production standard for most ingestion pipelines, because it preserves data and gives visibility into problems without stopping the pipeline. FAILFAST for regulatory feeds ensures that upstream schema violations are caught immediately, which is critical when the data

Definition

PERMISSIVE keeps bad rows with nulls and a _corrupt_record copy, DROPMALFORMED drops them, FAILFAST throws on the first bad record – tolerate, discard, or reject.

PERMISSIVE – the default mode; keeps the row, sets unparseable fields to null, and stores the raw bad record in a _corrupt_record column if that column is defined in the schema.
DROPMALFORMED – silently drops any row that does not conform to the schema, returning only clean rows.
FAILFAST – throws an exception on the first malformed record, aborting the entire read.
Choice criteria – tolerate (PERMISSIVE for Bronze layer), discard (DROPMALFORMED for known-dirty sources), or reject (FAILFAST for strict regulatory feeds).
_corrupt_record – must be explicitly added to the schema as a StringType; configurable via columnNameOfCorruptRecord.

When the National Bank of India imports a file of transactions and some lines are garbled, there are three policies. PERMISSIVE is the lenient clerk who files the garbled line anyway, leaving blanks where he could not read it, and keeps a photocopy of the original mess in a side folder. DROPMALFORMED is the clerk who quietly bins any unreadable line. FAILFAST is the strict clerk who stops the whole import and raises the alarm the moment he sees one bad line.

Understanding the Concepts

What are Spark read modes?

When Spark reads semi-structured files like CSV or JSON, some records may not match the expected schema. The mode option controls how Spark handles these malformed records. There are three choices: tolerate, discard, or reject.

PYTHON

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, DecimalType, DateType

spark = SparkSession.builder.appName("ReadModeDemo").getOrCreate()

# Define schema for transaction records at the National Bank of India
schema = StructType([
    StructField("txn_id", StringType()),
    StructField("account_id", StringType()),
    StructField("amount", DecimalType(12, 2)),
    StructField("txn_date", DateType()),
    StructField("_corrupt_record", StringType())  # For PERMISSIVE mode
])

How does PERMISSIVE mode work?

PERMISSIVE is the default and the most forgiving. A row with unparseable fields is kept, those fields become null, and if a _corrupt_record column is in the schema, the entire raw bad record is stored there.

PYTHON

# PERMISSIVE: keep everything, quarantine bad records
df_permissive = spark.read \
    .option("mode", "PERMISSIVE") \
    .option("columnNameOfCorruptRecord", "_corrupt_record") \
    .schema(schema) \
    .csv("/data/incoming/transactions.csv")

# Split clean rows from quarantined rows
clean = df_permissive.filter(df_permissive["_corrupt_record"].isNull()) \
    .drop("_corrupt_record")
quarantined = df_permissive.filter(df_permissive["_corrupt_record"].isNotNull())

print(f"Clean rows: {clean.count()}")
print(f"Quarantined rows: {quarantined.count()}")
quarantined.select("_corrupt_record").show(truncate=False)

TEXT

Clean rows: 9847
Quarantined rows: 153
+-----------------------------------------------+
|_corrupt_record                                 |
+-----------------------------------------------+
|TXN001,SAV2001,not_a_number,2024-01-15          |
|TXN002,,50000.00,invalid_date                   |
|malformed line with no commas at all            |
+-----------------------------------------------+

How does DROPMALFORMED mode work?

DROPMALFORMED silently removes any record that cannot be parsed to the schema. The result contains only clean rows and no error is raised.

PYTHON

# DROPMALFORMED: drop bad rows silently
df_dropped = spark.read \
    .option("mode", "DROPMALFORMED") \
    .schema(schema) \
    .csv("/data/incoming/transactions.csv")

# Only clean rows remain; bad rows are gone with no trace
print(f"Rows after dropping malformed: {df_dropped.count()}")
# WARNING: you have no visibility into what was dropped!

How does FAILFAST mode work?

FAILFAST throws an exception on the very first malformed record, stopping the entire read immediately. No data is returned.

PYTHON

# FAILFAST: abort on first bad record
try:
    df_strict = spark.read \
        .option("mode", "FAILFAST") \
        .schema(schema) \
        .csv("/data/regulatory/rbi_extract.csv")
    df_strict.count()  # Triggers the read
except Exception as e:
    print(f"Read aborted: {e}")
    # Alert the data quality team and halt the pipeline

TEXT

Read aborted: Malformed records are detected in record parsing.
SQL field "amount", field index 2: cannot cast "not_a_number" to DecimalType.

When do you use each mode?

The choice depends on how the pipeline should treat bad data.

| Mode | Behavior | Use Case | Risk |
|---|---|---|---|
| PERMISSIVE | Keep all, nullify bad fields, quarantine | Bronze layer ingestion | Must monitor quarantine |
| DROPMALFORMED | Drop bad rows silently | Known-dirty sources where loss is OK | Silent data loss |
| FAILFAST | Abort on first bad record | Regulatory feeds, strict pipelines | Blocks the whole load |

PYTHON

# Production pattern: PERMISSIVE with quarantine and alerting
def ingest_with_quarantine(spark, path, schema):
    df = spark.read \
        .option("mode", "PERMISSIVE") \
        .option("columnNameOfCorruptRecord", "_corrupt_record") \
        .schema(schema) \
        .csv(path)

    clean = df.filter(df["_corrupt_record"].isNull()).drop("_corrupt_record")
    bad = df.filter(df["_corrupt_record"].isNotNull())

    bad_count = bad.count()
    total_count = df.count()
    bad_ratio = bad_count / total_count if total_count > 0 else 0

    if bad_ratio > 0.05:  # More than 5% bad records
        raise Exception(f"Bad record ratio {bad_ratio:.2%} exceeds threshold")

    bad.write.mode("append").parquet("/data/quarantine/transactions")
    return clean

What are the gotchas with _corrupt_record?

The _corrupt_record column has several practical quirks that catch people in production.

PYTHON

# GOTCHA 1: _corrupt_record must be in the schema explicitly
# If you omit it, PERMISSIVE still nullifies bad fields but does NOT
# save the raw text, so you lose visibility into what went wrong.

schema_without_corrupt = StructType([
    StructField("txn_id", StringType()),
    StructField("amount", DecimalType(12, 2)),
])
df = spark.read.option("mode", "PERMISSIVE").schema(schema_without_corrupt) \
    .csv("/data/incoming/transactions.csv")
# Bad rows appear as rows with all nulls; raw text is lost

# GOTCHA 2: selecting only _corrupt_record without other columns
# can fail due to an evaluation quirk; cache or write first
df.cache()
df.filter(df["_corrupt_record"].isNotNull()).show()
df.unpersist()

How does read mode interact with different file formats?

The mode option applies to CSV and JSON where records can be malformed. Parquet and ORC have a strict binary schema and do not produce corrupt records in the same way.

PYTHON

# CSV: mode applies to field parsing
df_csv = spark.read.option("mode", "FAILFAST").schema(schema) \
    .csv("/data/transactions.csv")

# JSON: mode applies to JSON parsing
df_json = spark.read.option("mode", "PERMISSIVE").schema(schema) \
    .json("/data/transactions.json")

# Parquet: schema is embedded; mode option has no effect
df_parquet = spark.read.parquet("/data/transactions.parquet")
# Parquet files either match the schema or fail structurally

What should a senior engineer know?

PERMISSIVE with alerting is the production standard for most ingestion pipelines, because it preserves data and gives visibility into problems without stopping the pipeline.
FAILFAST for regulatory feeds ensures that upstream schema violations are caught immediately, which is critical when the data drives regulatory reporting at the National Bank of India.
DROPMALFORMED should always be paired with monitoring of the drop rate, because a sudden increase in dropped records indicates a systematic upstream change.
Schema evolution – when a source adds new columns, PERMISSIVE handles it gracefully by nullifying the unknown fields, while FAILFAST would break.
Performance – PERMISSIVE has a small overhead for maintaining the corrupt record column, but it is negligible compared to the I/O cost of reading the files.

Visual Overview

DIAGRAM

  Input File: transactions.csv
  ┌────────────────────────────────────────────┐
  │ TXN001,SAV2001,50000.00,2024-01-15  (good) │
  │ TXN002,CUR3001,not_a_num,2024-01-16 (bad)  │
  │ TXN003,SAV2001,25000.00,2024-01-17  (good) │
  │ malformed line no commas             (bad)  │
  └──────────────────┬─────────────────────────┘
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
  PERMISSIVE    DROPMALFORMED   FAILFAST
  ┌──────────┐  ┌──────────┐  ┌──────────┐
  │ 4 rows   │  │ 2 rows   │  │ EXCEPTION│
  │ 2 clean  │  │ (clean   │  │ on row 2 │
  │ 2 with   │  │  only)   │  │ (aborted)│
  │ nulls +  │  │          │  │          │
  │ _corrupt │  │ Bad rows │  │ No data  │
  │ _record  │  │ silently │  │ returned │
  │          │  │ dropped  │  │          │
  └──────────┘  └──────────┘  └──────────┘

Key Takeaway

Spark offers three read modes for handling malformed records in semi-structured files like CSV and JSON. PERMISSIVE is the default and the most forgiving: it keeps the row, sets unparseable fields to null, and if you include a _corrupt_record column in the schema, stores the raw bad record there for inspection. DROPMALFORMED silently drops any row that cannot be parsed and returns only clean rows with no error. FAILFAST throws an exception on the very first malformed record and aborts the entire read.

The choice depends on how the pipeline should treat bad data. PERMISSIVE with a _corrupt_record column is the standard for Bronze layer ingestion, where you want to preserve everything and quarantine problems for the data quality team rather than losing data or blocking the pipeline. FAILFAST is for strict regulatory feeds, where a single malformed record signals an upstream problem that must be fixed before anything downstream runs. DROPMALFORMED is the most dangerous because it silently loses data, so it should always be paired with monitoring of the drop rate.

At the National Bank of India, the general transactions feed used PERMISSIVE with quarantine and alerting: bad records went to a review table and the pipeline raised an alert if the bad-record ratio exceeded a threshold. For the RBI regulatory extract, FAILFAST ensured that any schema violation stopped the load immediately. The practical note to always remember is that PERMISSIVE requires explicitly adding the _corrupt_record column to the schema, and you should cache or write before filtering on it alone due to an evaluation quirk.

Remember one sentence: PERMISSIVE tolerates and quarantines, DROPMALFORMED discards silently, and FAILFAST rejects immediately – choose based on whether bad data should be preserved, ignored, or treated as a pipeline-stopping error.

---