Databricks & Delta Lake

Delta Live Tables is declarative ETL where you define tables, transformations, and quality expectations, and the framework auto-manages dependencies, execution, and monitoring, while Structured Streaming is the lower-level imperative engine.

In Databricks, CDC keeps a target in sync with a source's changes, using Delta MERGE INTO for batch upserts and DLT APPLY CHANGES INTO for streaming change feeds.

All-Purpose clusters are shared and interactive for development, Job clusters are ephemeral and isolated for production jobs, and SQL Warehouses are tuned endpoints for SQL and BI.

Time travel reads any past version of a Delta table by version number or timestamp, because every commit is kept as an immutable snapshot.

MERGE INTO is a single atomic upsert that matches source to target and applies UPDATE, DELETE, and INSERT together, safely under concurrency.

OPTIMIZE compacts small files and can ZORDER for faster reads, while VACUUM permanently deletes old unreferenced files to save storage, and vacuuming below the retention window breaks time travel.

Unity Catalog is Databricks' single, centralized governance layer for access control, auditing, lineage, and discovery across all workspaces and clouds, organized as catalog.schema.table.