Definition
- Bronze – the raw landing zone; data is ingested with minimal transformation, append-only, preserving the original bytes plus lineage metadata like ingestion timestamp and source file.
- Silver – the cleaned layer; types are cast, schemas enforced, corrupt records filtered, duplicates removed, and values conformed (e.g., normalized account IDs).
- Gold – the consumption layer; business-level aggregates, features, and marts shaped for reports, dashboards, and models.
- Rebuildability – each layer is a deterministic function of the layer below it plus code, so any layer can be rebuilt by reprocessing, making the pipeline idempotent and auditable.
Think of the National Bank's data as mail. Bronze is the mailroom where every letter is dropped in exactly as it arrived, nothing thrown away. Silver is the sorting office where letters are opened, duplicates removed, addresses corrected, and junk discarded. Gold is the manager's desk where clean letters are summarized into a daily report he can act on. If the manager's report is wrong, he does not panic – he re-sorts from the mailroom, because the raw mail is always kept.
Understanding the Concepts
What is the Bronze layer?
Bronze is the landing zone. You ingest source data with as little transformation as possible, just adding metadata like ingestion timestamp and source file name. It is append-only so you never lose history.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("MedallionDemo").getOrCreate()
# Ingest raw CSV from the vendor with minimal transformation
raw = spark.read.csv("/landing/daily_trades/2026-10-30/", header=True)
bronze = raw.withColumn("_ingest_ts", F.current_timestamp()) \
.withColumn("_source_file", F.input_file_name())
bronze.write.format("delta") \
.mode("append") \
.save("/lake/bronze/trades")
Bronze is your insurance policy: if any downstream logic has a bug, you still have the original bytes.
What is the Silver layer?
Silver applies the real engineering. You parse and cast types, enforce the schema, filter corrupt records, deduplicate on business keys, and conform values.
from pyspark.sql.window import Window
bronze = spark.read.format("delta").load("/lake/bronze/trades")
# Cast types
silver = bronze.select(
F.col("trade_id").cast("long"),
F.col("account_id").cast("long"),
F.col("amount").cast("decimal(18,2)"),
F.to_date("trade_date", "yyyy-MM-dd").alias("trade_date"),
F.col("branch_code"),
F.col("_ingest_ts")
)
# Remove corrupt records
silver = silver.filter(
F.col("trade_id").isNotNull() &
(F.col("amount") > 0)
)
# Deduplicate on business key, keeping the latest ingestion
window = Window.partitionBy("trade_id").orderBy(F.col("_ingest_ts").desc())
silver = silver.withColumn("rn", F.row_number().over(window)) \
.filter("rn = 1") \
.drop("rn")
silver.write.format("delta") \
.mode("overwrite") \
.save("/lake/silver/trades")
Silver is where data quality lives, so downstream consumers can trust it.
What is the Gold layer?
Gold is consumption-shaped. You aggregate to business grain, join reference data, and compute features or KPIs.
silver = spark.read.format("delta").load("/lake/silver/trades")
branches = spark.read.format("delta").load("/lake/dim/branches")
# Daily balance per account
gold_daily = silver.groupBy("account_id", "trade_date").agg(
F.sum("amount").alias("daily_total"),
F.count("*").alias("trade_count")
)
# Branch summary with enrichment
gold_branch = silver.join(branches, "branch_code") \
.groupBy("branch_name", "trade_date") \
.agg(F.sum("amount").alias("branch_total"))
gold_daily.write.format("delta").mode("overwrite") \
.save("/lake/gold/daily_account_totals")
gold_branch.write.format("delta").mode("overwrite") \
.save("/lake/gold/branch_summary")
These tables are small, fast, and tailored to a specific use case.
Why does rebuildability give you reliability?
Each layer is a pure, deterministic function of the layer below plus code. If Gold is wrong, fix the code and rerun from Silver. If Silver is wrong, rerun from Bronze.
# Bug found in Gold aggregation – fix and rebuild
# Step 1: fix the code
# Step 2: rerun Gold from intact Silver
gold_daily = spark.read.format("delta").load("/lake/silver/trades") \
.groupBy("account_id", "trade_date") \
.agg(F.sum("amount").alias("daily_total")) # fixed formula
gold_daily.write.format("delta").mode("overwrite") \
.save("/lake/gold/daily_account_totals")
# No need to re-fetch from fragile source systems
# Bronze is the immutable replay buffer
This makes pipelines idempotent and reproducible, which is essential in banking where auditors ask you to reproduce a number from months ago.
How does the medallion architecture handle schema evolution?
Bronze preserves the raw format. Silver normalizes schemas. When the source adds a column, Bronze captures it and Silver adds the cast.
# Source vendor adds a new "fee" column
# Bronze: automatically captured because we ingest all columns
# Silver: add the new column cast
silver = bronze.select(
F.col("trade_id").cast("long"),
F.col("account_id").cast("long"),
F.col("amount").cast("decimal(18,2)"),
F.col("fee").cast("decimal(18,2)"), # new column
F.to_date("trade_date", "yyyy-MM-dd").alias("trade_date"),
F.col("branch_code")
)
# With Delta Lake schema evolution:
silver.write.format("delta") \
.option("mergeSchema", "true") \
.mode("overwrite") \
.save("/lake/silver/trades")
How do you separate concerns and permissions?
The layers provide natural permission boundaries.
# Bronze: raw PII, restricted access
# - Only the ingestion service and data engineers can read
# - Contains full customer names, addresses, phone numbers
# Silver: PII masked or tokenized
silver = silver.withColumn(
"holder_name",
F.concat(F.substring("holder_name", 1, 1), F.lit("*****"))
)
# Gold: aggregates only, no PII
# - Analysts and dashboards access Gold
# - No individual customer data exposed
What role does Delta Lake play?
Delta Lake adds ACID transactions, time travel, and schema enforcement that make the medallion pattern robust.
# Time travel: reproduce any historical state
historical = spark.read.format("delta") \
.option("versionAsOf", 5) \
.load("/lake/gold/daily_account_totals")
# ACID: concurrent writes to Bronze are safe
# Schema enforcement: Silver rejects data that does not match
What should a senior engineer know?
- Bronze is immutable and append-only; never delete or overwrite raw data.
- Silver deduplication must be idempotent so reruns produce the same result.
- Gold tables should be narrow and query-specific; a single Gold table per use case.
- Delta Lake time travel provides point-in-time reproducibility for auditors.
- The medallion pattern works with streaming: Auto Loader ingests to Bronze, Delta Live Tables process Silver and Gold.
# Streaming medallion with Auto Loader
bronze_stream = spark.readStream \
.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.load("/landing/trades/")
bronze_stream.writeStream \
.format("delta") \
.option("checkpointLocation", "/checkpoints/bronze_trades") \
.start("/lake/bronze/trades")
What are common interview follow-up questions?
- Can you have more than three layers? Yes, some teams add a "raw" layer before Bronze or a "platinum" layer for ML features, but three is the standard.
- What if Bronze data is corrupted at the source? Bronze faithfully stores the corruption. Silver's validation catches and quarantines bad records.
- How long do you keep Bronze data? As long as compliance and storage costs allow. In banking, often 7+ years.
Visual Overview
Medallion Architecture Flow
───────────────────────────
Source Systems
│
│ ingest (minimal transform)
▼
┌──────────────────────────────────┐
│ BRONZE (raw, append-only) │
│ - original bytes preserved │
│ - _ingest_ts, _source_file │
│ - insurance policy │
└──────────────┬───────────────────┘
│ cast, validate, dedup
▼
┌──────────────────────────────────┐
│ SILVER (clean, conformed) │
│ - proper types and schema │
│ - duplicates removed │
│ - corrupt records filtered │
└──────────────┬───────────────────┘
│ aggregate, join, compute
▼
┌──────────────────────────────────┐
│ GOLD (business-ready) │
│ - KPIs, features, marts │
│ - small, fast, query-specific │
│ - consumed by dashboards, ML │
└──────────────────────────────────┘
Rebuild: Gold ← Silver ← Bronze (always possible)
Key Takeaway
The Medallion architecture organizes a data pipeline into Bronze, Silver, and Gold layers, usually on Delta Lake. Bronze is the raw landing zone, append-only, holding data exactly as received with lineage metadata. Silver is the cleaned and conformed layer where types are cast, schemas enforced, duplicates removed, and values normalized. Gold is the business layer holding aggregates, features, and marts shaped for reports, dashboards, and models.
The key engineering property is that each layer is a deterministic function of the layer below it plus code. That gives reliability and reproducibility. If a Gold number is wrong, fix the code and rebuild Gold from Silver. If Silver is wrong, rebuild it from Bronze. You never need to go back to fragile source systems because Bronze is your immutable replay buffer. This makes the pipeline idempotent and makes backfills and corrections routine.
At the National Bank of India, Bronze kept every trade file the vendor sent, Silver deduplicated on trade id and filtered malformed records, and the risk team consumed a Gold table of daily spend per account. When a bug once produced wrong fraud totals, we did not scramble to re-fetch data – we fixed the aggregation logic and reran only the Gold step from an intact Silver table. The layers also provided natural permission boundaries: raw PII stayed in Bronze, Silver applied masking, and Gold exposed only aggregates.
Remember one sentence: Bronze keeps the raw truth, Silver makes it clean, Gold makes it useful, and because every layer rebuilds from the one below, the pipeline is reproducible enough to satisfy a banking auditor.
---