Apache Spark: Core

Storage levels control how cached partitions are stored, trading speed, memory, durability, and CPU: memory-only for speed, memory-and-disk for safety, replicated for fault tolerance, off-heap to avoid GC.

Cache when a dataset is both expensive to produce and reused several times; do not cache single-use data, trivially cheap data, or so much data that eviction and memory pressure hurt the whole job.

unpersist() promptly frees a cache's memory and disk, which is essential in long jobs to prevent storage exhaustion, and the Spark UI Storage tab shows cached partitions and memory used to verify effectiveness.

Cache keeps data on executors for fast reuse and preserves lineage, while checkpoint writes data to reliable storage and truncates lineage for durable fault recovery.

StructType is a nested object, ArrayType is a list, MapType is a dictionary; access with dot notation, brackets, getItem, and use explode to flatten into rows.

Post-shuffle coalescing starts with many fixed shuffle partitions, then merges the small ones at runtime into fewer right-sized partitions so the next stage runs fewer, meaningful tasks.

Flink SQL is true continuous streaming with native changelog semantics and millisecond latency; Spark SQL streaming is micro-batch by default with a unified batch-stream API and a larger ecosystem.

Broadcast for a small side, sort-merge for two large sides, shuffle-hash when one fits per partition, and nested loop for non-equi joins: the optimizer picks by size, threshold, and join condition.

A Broadcast Join copies the small table to every executor so the large table joins locally with no shuffle: in PySpark, wrap the small side in broadcast(small).

The threshold (default 10 MB) auto-broadcasts any side Spark estimates at or below it; use an explicit hint when the estimate is wrong, stale, or just over the line.

Confirm a broadcast join by seeing BroadcastExchange plus BroadcastHashJoin in explain output and no shuffle on the large side in the Spark UI.

Write /*+ BROADCAST(alias) */ right after SELECT to force Spark to broadcast that table, overriding the automatic threshold.

AQE flips a shuffle join to a broadcast join once it sees a side is actually small after earlier stages run, and the local shuffle reader lets executors reuse already-written shuffle data locally.

Caching stores computed partitions on executors so later actions skip recomputation, and cache() is lazy because it only marks intent and the data is stored on the first action.

cache() is persist() with a fixed default storage level, while persist(StorageLevel) lets you explicitly choose memory vs disk, serialized vs deserialized, off-heap, and replication.

The Driver plans and schedules work, the Cluster Manager allocates resources, Executors on Worker Nodes run Tasks in parallel, and one Task processes one partition.

An RDD is an immutable, partitioned, lazily evaluated, lineage-recoverable distributed collection, and its immutability is what makes parallelism safe and recomputation deterministic.

Plain Python UDFs are slow because every row crosses the JVM-to-Python boundary and the optimizer cannot see inside them: prefer built-ins, then Pandas UDFs.

Because the same key always hashes to the same bucket, bucketing lets Spark read one bucket for equality filters, aggregate per bucket without shuffle, and join bucket-to-bucket without shuffle.

No: Spark is a YARN application that ships its own jars to containers, so install it only on the edge node, not on every worker.

Cluster mode puts the Driver inside the cluster for robust production runs, client mode keeps it on your machine for interactive work, and local mode runs everything in one JVM for development and testing.

Spark is a distributed computing engine that processes large datasets in parallel across a cluster, keeping data in memory so it runs far faster than disk-based MapReduce.

Parallelize JDBC reads with a partition column and bounds, tune fetchsize for throughput, push filters into SQL, and design writes for idempotency because each task is its own transaction.

Flink is streaming-first with true record-at-a-time processing and richer state, while Spark Structured Streaming is batch-first with micro-batch, a unified API, and a larger ecosystem.

CSV inference reads the data twice, JSON infers from a sample, Parquet/ORC carry the schema in the footer so no inference is needed, and an explicit schema in production avoids the double read and prevents schema drift.

Spark sort is not stable because parallel sorting reorders equal keys, so add a unique tie-breaker column when you need deterministic ordering.

The Block Manager is the per-node storage engine for blocks across memory, disk, and off-heap, and a master registry locates blocks, powering caching, broadcasts, shuffle, and temp storage.

Execution and Storage share one unified region with a movable boundary; either can borrow the other's idle space, but execution has priority and can evict borrowed cached blocks, while active execution memory is never evicted.

Debug a Spark OOM by finding whether the driver or an executor died and why, using the UI's Stages, Executors, Storage, and SQL tabs to spot skew, excessive shuffle, oversized broadcast, too few partitions, or a driver-side collect.

Spill-to-disk writes overflow shuffle/sort/aggregation data to local disk to avoid OOM; it keeps the job correct but, because disk is far slower than memory, heavy spilling is a red flag.

`union` lines up columns by position and silently corrupts data when orders differ, while `unionByName` lines them up by name and is the safe default.

An RDD is fully described by its partitions, its per-partition compute function, its parent dependencies (lineage), an optional partitioner, and optional preferred locations for data locality.

Session start, executor provisioning, DAG build, Catalyst optimization, stage splitting at shuffles, one task per partition, parallel execution, shuffle exchange, results out, and session stop – that is the full Spark lifecycle.

`repartition(n)` fully shuffles into n evenly-sized partitions and can increase or decrease, while `coalesce(n)` merges existing partitions with no shuffle, only decreases, and can be uneven.

`repartition(n)` balances evenly, `repartition(cols)` groups by key, and `repartition(n, cols)` does both into a fixed count – all three pay for a full shuffle.

`coalesce(n, shuffle=true)` turns the cheap no-shuffle merge into a full shuffle, which is exactly how `repartition(n)` is built.

Coalesce merges cheaply but only shrinks; repartition shuffles to both grow and balance – so reduce with coalesce, increase with repartition.

Repartition after a heavy filter to rebalance skewed partitions; coalesce before writing to avoid a flood of tiny files.

`partitionBy` writes data into value-named directories so queries filtering on that column read only the matching directories and skip everything else.

Static pruning uses a literal known at plan time; dynamic pruning derives the pruning values at runtime from the other side of a join.

Partition by low-cardinality columns you actually filter on; partitioning on a high-cardinality column explodes into millions of tiny directories and crashes planning.

Bucketing hashes a column into a fixed number of files so the same key always lands in the same bucket, giving join and aggregation benefits without the directory explosion of partitioning.

Speculative execution relaunches straggler tasks on other executors and takes whichever finishes first; enable it for environmental slowness, disable it when the cause is data skew.

Dynamic allocation scales executors up on task backlog and down on idle timeout, but requires an external shuffle service or shuffle tracking so removing an executor does not lose its shuffle files.

void (NullType) is the untyped-null type from lit(None); typed formats reject it, so cast it to a concrete type like lit(None).cast('string').

HashAggregate uses a fast off-heap hash map of mutable fixed-length buffers; SortAggregate sorts first for buffers that are not fixed-length; ObjectHashAggregate handles object buffers like collect_list; Spark picks by buffer type.

SparkContext is the low-level RDD and cluster connection (one per JVM); SparkSession is the modern unified front door that wraps it and adds DataFrames, SQL, and catalog.

Point Spark at the Hive metastore via hive-site.xml, turn on enableHiveSupport, set the warehouse dir, and Spark can read, write, and query Hive tables with HiveQL on Spark's engine.

Data skew is one partition holding far more data than the rest due to a hot key, creating a straggler task – spot it in the Spark UI as one task taking ten to a hundred times longer with lopsided shuffle sizes.

Salting appends a random suffix to the hot key to spread it across N partitions and replicates the other side N times to still match – trading duplicated data for balanced partitions.

Non-deterministic functions (rand, uuid, monotonically_increasing_id, shuffle, input_file_name, spark_partition_id, order-sensitive collect_list) return different results across runs, so Catalyst will not push them past filters or freely reorder and re-execute them.

Window functions compute per-row values over a PARTITION BY and ORDER BY window – ranking, LAG/LEAD, running sums, moving averages – and unlike groupBy they annotate every row instead of collapsing groups into one summary row.

PERMISSIVE keeps bad rows with nulls and a _corrupt_record copy, DROPMALFORMED drops them, FAILFAST throws on the first bad record – tolerate, discard, or reject.

OPTIMIZE compacts small files, ZORDER clusters them for skipping, min/max stats let Delta skip files, and VACUUM reclaims dead files.

Choose Snappy or ZSTD for a good speed-and-ratio balance and splittability, avoid Gzip for large splittable reads, and remember output-format compression and internal spark.io.compression.codec are two separate settings.

A skipped stage is one Spark reused instead of recomputing because its output already existed (from prior shuffle output, cache, or checkpoint), common across successive actions on the same cached DataFrame.

A stage is a shuffle-free run of work, the DAGScheduler cuts stages at every wide dependency, and within each stage there is exactly one task per partition.

Spark is a fast in-memory distributed engine built to beat MapReduce's disk I/O, offering SQL, streaming, machine learning, and graph libraries on one unified engine.

Barrier Execution Mode launches all tasks of a stage together or not at all, and restarts the whole stage on any single failure, so that MPI-style frameworks like distributed TensorFlow or PyTorch can run their all-to-all synchronized training on Spark.

`cogroup` bundles all values from two RDDs per key into two separate iterables, giving you both sides of a key together without the row multiplication a join causes.

Predicate pushdown filters at the source before data enters Spark, saving I/O – powerful for columnar Parquet/ORC with statistics, weak for row-based CSV/JSON that must read every row.

Py4J is the Python-to-JVM socket bridge in PySpark, and because Python UDFs run outside the JVM, every row must be serialized across that bridge and back, which is the overhead.

Secure a production Spark cluster across four layers – authentication (Kerberos/tokens), authorization (Ranger/RBAC/ACLs), encryption at rest and in transit (SSL/TLS for RPC and shuffle), and network isolation – defense in depth, not any single control.

Spark serializes all shuffle, broadcast, closure, and serialized cache data; Java serialization is the slow, bulky default, while Kryo (spark.serializer=KryoSerializer) is faster and smaller, especially after registering your classes.

distinct() dedupes on all columns (whole-row match); dropDuplicates(subset) dedupes on just the chosen key columns and keeps the first occurrence – use the subset form when a business key defines uniqueness.

groupByKey shuffles all raw values then aggregates; reduceByKey pre-aggregates on each partition before shuffling, so it moves far less data – prefer reduceByKey for associative aggregations.

Too many tiny files cause metadata, listing, and scheduling overhead; fix it with coalesce before write, maxRecordsPerFile, periodic compaction, or Delta Lake OPTIMIZE to produce fewer, right-sized files.

Spark DataFrames are distributed and lazy, building an optimized plan that only runs on an action; Pandas DataFrames are single-machine and eager, running every step immediately.

The Spark Thrift Server exposes Spark SQL as a JDBC/ODBC endpoint so standard BI tools run SQL on a shared, always-on Spark cluster without any Spark dependency.

The Spark UI (port 4040 live, History Server for completed apps) debugs through Jobs, Stages, SQL, Storage, and Executors tabs – one task far slower means skew, high GC time means memory pressure, and large shuffle write means an expensive wide transformation.

_SUCCESS is an empty marker file written only after a Spark write fully commits, so downstream workflows know the output is complete and safe to consume.

Ingest through a partitioned durable log, process as streaming or batch to fit the need, and store in columnar format, deciding partitioning, semantics, mode, and reliability deliberately.