Advanced Data Engineering

Deserialized storage is fastest but memory-hungry because it keeps live objects, while serialized storage packs data into compact bytes to save memory at the cost of serialization CPU.

Do not cache everything, do not assume cache is eager, do not cache single-use data, do not forget unpersist, do not confuse cache with collect, and do not expect cache to remove a shuffle.

explode drops rows with null or empty arrays, explode_outer keeps them with nulls, and you filter and select before exploding because every parent row and column gets multiplied.

Higher-order array functions (transform, filter, aggregate, exists, forall) process array elements in place with a lambda, avoiding the explode-then-regroup cost.

Auto Loader incrementally ingests only the new files arriving in cloud storage, exactly once, using checkpointing plus directory listing or file notifications, so it scales to millions of files without reprocessing.

AQE coalesces tiny partitions into balanced ones, splits oversized skewed partitions to kill stragglers, and switches a shuffle join to a broadcast join when a side turns out small.

Know the master, skew, and coalesce switches plus the key thresholds, and avoid the three traps: assuming AQE is on, tuning blindly, and expecting it to cure extreme skew.

A Pandas UDF processes a whole column at once via Arrow batches as a pandas Series, turning per-row overhead into per-batch overhead.

Kinesis Data Analytics integrates with Lambda at input for pre-processing and at output for serverless custom actions, creating a fully serverless event-driven pipeline.

UnsafeRow is a flat binary row in raw memory that avoids serialization overhead, and it is a core pillar of Project Tungsten's fast execution engine.

insertInto matches by position, so wrong column order silently writes wrong data, and a high-cardinality column in the partition position creates millions of tiny partitions.

Narrow dependencies keep each child partition fed by one parent partition with no shuffle, while wide dependencies spread one parent partition across many children and force an expensive network shuffle.

Spark recovers from failures by replaying the recorded lineage recipe to recompute only the lost partitions, not by replicating data or rerunning the whole job.

Read less data – prune partitions, cover the query with the right composite index, pre-aggregate repeated rollups, and rewrite to avoid full scans – and justify each change from the query plan.

Temp views are session-scoped names for DataFrames; global temp views are app-wide via global_temp; managed tables let Spark own data and metadata so drop deletes files, while external tables let Spark own only metadata so drop keeps the files.

MSK is managed open-source Kafka with full ecosystem portability, while Kinesis is simpler AWS-native streaming with deep integration; choose based on compatibility needs vs operational simplicity.

collect pulls all rows to the Driver and can crash it; take(n) and head(n) safely pull only n; first() is take(1) – never collect a large DataFrame.

GraphX does graph-parallel computation on vertices and edges (PageRank, connected components, triangle counting); MLlib does distributed machine learning (classification, regression, clustering, recommendation) with a Pipeline API.

MSCK REPAIR TABLE makes the metastore catch up with partition folders that already exist on disk but were added outside the SQL engine.

pathGlobFilter keeps only files whose names match a glob pattern, while partition discovery turns folder-name structure into columns for pruning; one filters files, the other derives partition metadata.

Projection pushdown reads only the columns the query needs, so in columnar storage selecting five of a hundred columns reads about five percent of the data.

SizeEstimator weighs a dataset's real in-memory byte footprint so you can decide whether it is small enough to broadcast to every executor safely.

Bronze keeps the raw truth, Silver makes it clean, Gold makes it useful, and every layer can be rebuilt from the one below.

withColumn in a loop re-analyzes a deepening plan each iteration (O(n-squared) on the Driver); add all columns at once with a single select or withColumns.