Definition
Delta Live Tables raises the abstraction above Structured Streaming.
- Declarative tables – You define each table as a decorated function returning a DataFrame; the framework resolves dependencies automatically.
- Quality expectations –
@dlt.expectkeeps bad rows and records a metric;@dlt.expect_or_dropremoves them;@dlt.expect_or_failhalts the pipeline. - Automatic dependency management – DLT builds a dependency graph from
dlt.readanddlt.read_streamreferences and runs tables in topological order. - Built-in monitoring – Visual lineage graph, run history, quality dashboards, and automatic retries.
- Structured Streaming underneath – DLT manages checkpoints, incremental processing, and streaming state internally.
Use DLT for standard multi-stage ETL; drop to Structured Streaming for custom stateful logic.
Imagine the National Bank of India needs a multi-step assembly line that turns raw trade records into clean records and then into a fraud report. With Structured Streaming you are the factory engineer who personally builds each conveyor belt, wires the motors, and installs the quality inspectors. With Delta Live Tables you write a blueprint saying raw feeds into clean, clean feeds into the report, and here are the quality rules, and the framework builds the conveyors, runs them in the right order, and self-heals when something jams.
Understanding the Concepts
What is Delta Live Tables?
Delta Live Tables (DLT) is a declarative ETL framework on Databricks where you define the tables you want and the transformations that produce them, and the framework handles dependency ordering, execution, checkpointing, and monitoring.
import dlt
from pyspark.sql.functions import col
# Bronze: ingest raw trade records
@dlt.table(
comment="Raw trade records from Kafka"
)
def bronze_trades():
return (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/data/bank/landing/trades/")
)
How do quality expectations work?
Expectations are declarative quality rules attached to tables. They come in three flavors: expect (keep and record), expect_or_drop (remove bad rows), and expect_or_fail (halt the pipeline).
# Silver: clean and validate trades
@dlt.table(
comment="Cleaned trade records with quality checks"
)
@dlt.expect("valid_trade_id", "trade_id IS NOT NULL")
@dlt.expect_or_drop("valid_amount", "amount > 0")
@dlt.expect_or_fail("valid_account", "account_id IS NOT NULL")
def silver_trades():
return (
dlt.read_stream("bronze_trades")
.select(
col("trade_id"),
col("account_id"),
col("amount").cast("decimal(18,2)"),
col("trade_date").cast("date"),
col("branch_code")
)
)
Expectation behavior:
expect("valid_trade_id") → keeps bad rows, records violation as metric
expect_or_drop("valid_amount") → drops rows where amount <= 0
expect_or_fail("valid_account") → stops pipeline if any account_id is NULL
Pipeline UI shows:
valid_trade_id: 99.8% pass (20 violations)
valid_amount: 99.99% pass (5 rows dropped)
valid_account: 100% pass
How does automatic dependency management work?
DLT builds a dependency graph from dlt.read and dlt.read_stream references. You never manually sequence stages: the framework resolves the topological order.
# Gold: aggregate by branch (depends on silver_trades)
@dlt.table(
comment="Daily trade totals by branch"
)
def gold_branch_totals():
return (
dlt.read("silver_trades")
.groupBy("branch_code", "trade_date")
.agg({"amount": "sum", "trade_id": "count"})
.withColumnRenamed("sum(amount)", "total_amount")
.withColumnRenamed("count(trade_id)", "trade_count")
)
# DLT automatically resolves:
# bronze_trades → silver_trades → gold_branch_totals
Dependency graph (auto-resolved):
bronze_trades ──► silver_trades ──► gold_branch_totals
│ │
└── Auto Loader ──────────────────────┘
(checkpoint managed by DLT)
How does DLT differ from Structured Streaming?
Structured Streaming is the lower-level imperative engine. With it, you write each stream, set up each checkpoint, decide execution order, and add validation manually. DLT raises the abstraction.
| Feature | Structured Streaming | Delta Live Tables |
|---|---|---|
| Programming model | Imperative | Declarative |
| Dependency ordering | Manual | Automatic |
| Checkpoint management | Manual | Automatic |
| Quality checks | Custom code | Built-in EXPECT |
| Monitoring UI | Build your own | Built-in lineage + dashboards |
| Recovery | Manual restart | Auto-retry |
| Control level | Full | Framework-managed |
# Structured Streaming: manual wiring
# You must manage each stream, checkpoint, and ordering yourself
raw_stream = spark.readStream.format("json").load("/data/bank/landing/")
clean_stream = raw_stream.filter("amount > 0 AND account_id IS NOT NULL")
clean_stream.writeStream \
.format("delta") \
.option("checkpointLocation", "/checkpoints/silver_trades") \
.outputMode("append") \
.start("/data/bank/silver/trades")
# DLT: same pipeline, declarative
# No checkpoint management, no manual ordering, built-in quality
How does DLT support the medallion architecture?
DLT naturally underpins the bronze-silver-gold medallion architecture: each layer is just a DLT table that reads from the one below.
# Complete medallion pipeline in DLT
@dlt.table(comment="Bronze: raw ingestion")
def bronze_trades():
return spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "json") \
.load("/data/bank/landing/trades/")
@dlt.table(comment="Silver: cleaned and validated")
@dlt.expect_or_drop("positive_amount", "amount > 0")
def silver_trades():
return dlt.read_stream("bronze_trades") \
.select("trade_id", "account_id", "amount", "trade_date", "branch_code")
@dlt.table(comment="Gold: business aggregates")
def gold_branch_totals():
return dlt.read("silver_trades") \
.groupBy("branch_code", "trade_date") \
.agg({"amount": "sum"})
How does DLT handle Auto Loader and CDC?
DLT integrates Auto Loader for ingestion and supports change data capture through APPLY CHANGES INTO.
# Auto Loader integration in DLT
@dlt.table(comment="Ingest new files automatically")
def bronze_accounts():
return (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "/data/bank/schema/accounts")
.load("/data/bank/landing/accounts/")
)
# CDC with APPLY CHANGES INTO
dlt.create_streaming_table("silver_accounts")
dlt.apply_changes(
target="silver_accounts",
source="bronze_cdc_events",
keys=["account_id"],
sequence_by=col("sequence_number"),
apply_as_deletes=col("operation") == "DELETE",
stored_as_scd_type=1 # Type 1: overwrite
)
What are streaming tables vs materialized views?
DLT supports two kinds of objects: streaming tables (process incrementally) and materialized views (recompute fully).
# Streaming table: incremental processing
@dlt.table(comment="Streaming: processes only new records")
def streaming_trades():
return dlt.read_stream("bronze_trades")
# Materialized view: full recomputation
@dlt.table(comment="Materialized view: recomputes on each run")
def branch_summary():
return dlt.read("silver_trades") \
.groupBy("branch_code") \
.agg({"amount": "sum", "trade_id": "count"})
What should a senior engineer know?
- DLT is a managed Databricks feature, so it adds platform dependence. It runs as its own pipeline with its own compute.
- Structured Streaming remains the right tool when you need custom stateful logic or you are embedding streaming inside a larger application.
- DLT expectations produce automatic pass/fail metrics visible in the pipeline UI, making data quality observable without custom dashboards.
- DLT supports triggered mode (run once, stop) and continuous mode (always running), configured at the pipeline level.
# Pipeline configuration (JSON)
pipeline_config = {
"name": "bank-trade-pipeline",
"target": "banking_prod",
"continuous": False, # Triggered mode: run and stop
"libraries": [
{"notebook": {"path": "/pipelines/trade_etl"}}
],
"clusters": [{
"autoscale": {"min_workers": 1, "max_workers": 5}
}]
}
Visual Overview
Delta Live Tables Pipeline:
Landing Zone DLT Pipeline (auto-managed)
┌─────────┐ ┌───────────────────────────────────┐
│ JSON │ │ │
│ files │────►│ bronze_trades │
│ (Auto │ │ (raw ingestion) │
│ Loader) │ │ │ │
└─────────┘ │ ▼ │
│ silver_trades │
│ EXPECT: amount > 0 │
│ EXPECT_OR_FAIL: account_id NOT NULL│
│ │ │
│ ▼ │
│ gold_branch_totals │
│ (aggregated by branch) │
│ │
│ ┌────────────────────────┐ │
│ │ Pipeline UI: │ │
│ │ - Lineage graph │ │
│ │ - Quality metrics │ │
│ │ - Run history │ │
│ │ - Auto-retry status │ │
│ └────────────────────────┘ │
└───────────────────────────────────┘
Key Takeaway
Delta Live Tables is a declarative ETL framework on Databricks where you define the tables you want and the transformations that produce them, and the framework figures out the dependency order, manages the underlying execution, enforces data-quality expectations, and gives you built-in monitoring and recovery. The contrast with Structured Streaming is the heart of the answer: Structured Streaming is the lower-level imperative engine where you write each stream yourself, set up each checkpoint, decide execution order, and add your own validation. DLT raises the abstraction so you declare the what, and the framework handles the how.
Its defining feature is quality expectations. You attach declarative rules to a table and choose an action: expect keeps bad rows and records a metric, expect_or_drop removes failing rows, and expect_or_fail stops the pipeline. These are first-class and tracked, so quality becomes part of the pipeline definition rather than scattered if-statements. DLT supports streaming tables and materialized views, naturally underpins the medallion architecture, and integrates Auto Loader and CDC.
At the National Bank of India our trade pipeline had three stages written as separate Structured Streaming jobs, and engineers had to manually ensure the order and wire checkpoints. We rewrote it in DLT as three declared tables: bronze reading the landing feed, silver reading bronze, and gold reading silver. The framework resolved the order automatically and managed the streaming plumbing. Quality expectations dropped negative amounts at the silver stage with automatic metrics, so bad data no longer reached the fraud report.
The honest nuance is that DLT uses Structured Streaming underneath and is a managed Databricks feature. Structured Streaming remains the right tool for custom stateful logic; DLT is the right tool for maintainable, quality-governed pipelines.
Remember one sentence: Delta Live Tables is declarative ETL with built-in quality expectations and automatic dependency management, sitting above Structured Streaming so you define the tables and the framework runs the pipeline.
---