Data Formats: Parquet, Avro, ORC

Parquet skips blocks using min/max ranges, dictionary membership, and bloom filters, and sorting the data makes min/max ranges tight enough to actually skip blocks.

Spark reads XML by naming the rowTag, and reads JDBC, Avro, Delta, Iceberg, and HBase through pluggable connectors, all via spark.read.format with format-specific options.

Avro is row-based with embedded schema for write-heavy ingestion; Parquet is columnar for read-heavy analytics; ORC is columnar with Hive ACID: use Avro to write, Parquet or ORC to read.

Transformations build a lazy plan and return a new dataset, actions execute the plan and return a result, and lazy evaluation lets Spark optimize the whole chain before running it.

Delta Lake is Parquet plus a transaction log that turns a dumb folder of files into an ACID, versioned, updatable table.

Parquet stores data by column so analytics reads only the columns it needs, skips irrelevant data, and compresses far better than row formats.