Definition
- Delta Lake – an open-source storage layer on top of Parquet that adds transactional capabilities to a data lake.
- Transaction log (_delta_log) – an ordered directory of JSON commit files plus periodic Parquet checkpoints that records every change to the table.
- ACID transactions – atomicity, consistency, isolation, and durability provided by the commit protocol.
- Schema enforcement – rejecting writes whose columns or types do not match the table definition.
- Time travel – querying the table as it was at a previous version or timestamp.
- MERGE – an atomic DML operation that inserts new rows and updates existing ones in one step.
Imagine the National Bank of India keeps all its transaction records as plain Parquet files in a folder. If Rajesh's transfer writes new files while a report is being generated, the report might read half-written data: there is no referee. Delta Lake adds a strict ledger (the transaction log) that records "at version 5, these files are the truth." Everyone reads the ledger first, so nobody ever sees a half-finished write, and you can ask "show me the table as it was at version 3 this morning."
Understanding the Concepts
What is wrong with plain Parquet?
Plain Parquet is just a columnar file format. A "Parquet table" is really a directory of .parquet files with no coordinator. When a writer adds or rewrites files, a concurrent reader can see a partial result, there is no way to atomically replace data, and there is no history.
package com.bank.formats;
public class PlainParquetProblems {
public static void main(String[] args) {
System.out.println("Plain Parquet problems:");
System.out.println(" 1. No ACID: concurrent reader sees"
+ " half-written files");
System.out.println(" 2. No atomic replace: partial update"
+ " visible");
System.out.println(" 3. No history: cannot query old versions");
System.out.println(" 4. No UPDATE/DELETE: must rewrite"
+ " entire dataset");
System.out.println(" 5. No schema enforcement: bad columns"
+ " land silently");
}
}
Plain Parquet problems:
1. No ACID: concurrent reader sees half-written files
2. No atomic replace: partial update visible
3. No history: cannot query old versions
4. No UPDATE/DELETE: must rewrite entire dataset
5. No schema enforcement: bad columns land silently
How does the Delta transaction log work?
Delta Lake adds the _delta_log folder containing numbered JSON commit files. Each commit is an atomic set of actions: add (new data file), remove (file logically deleted), and metaData (schema or partitioning changed). Readers reconstruct the table at version N by replaying commits 0 through N.
s3://bank-lake/curated/txns_delta/
_delta_log/
00000000000000000000.json (version 0: initial load)
00000000000000000001.json (version 1: append)
00000000000000000002.json (version 2: merge/update)
00000000000000000003.json (version 3: delete)
00000000000000000010.checkpoint.parquet (checkpoint)
part-00000-abc123.parquet
part-00001-def456.parquet
part-00002-ghi789.parquet
package com.bank.formats;
public class DeltaLogExplained {
public static void main(String[] args) {
System.out.println("Commit 0: add part-00000, part-00001");
System.out.println("Commit 1: add part-00002");
System.out.println("Commit 2: remove part-00000,"
+ " add part-00003");
System.out.println();
System.out.println("Table at version 2:");
System.out.println(" Active files: part-00001, part-00002,"
+ " part-00003");
System.out.println(" Removed: part-00000 (old, for time travel)");
}
}
Commit 0: add part-00000, part-00001
Commit 1: add part-00002
Commit 2: remove part-00000, add part-00003
Table at version 2:
Active files: part-00001, part-00002, part-00003
Removed: part-00000 (old, for time travel)
How does ACID work in Delta Lake?
Because each commit is a single atomic file creation, either the whole change is visible or none of it is. Concurrent writers use optimistic concurrency control: two writers prepare commits, race to create the next numbered log entry, and only one succeeds. The loser retries.
# Atomic append: either the whole commit is visible or not
df.write.format("delta").mode("append").save(
"s3://bank-lake/curated/txns_delta")
# If two writers race:
# Writer A creates 00000003.json -> succeeds
# Writer B tries 00000003.json -> fails (exists)
# Writer B re-reads state, retries as 00000004.json
package com.bank.formats;
public class DeltaACID {
public static void main(String[] args) {
System.out.println("ACID properties in Delta Lake:");
System.out.println(" Atomicity: commit is one file creation");
System.out.println(" Consistency: schema enforced per write");
System.out.println(" Isolation: readers see committed versions");
System.out.println(" Durability: log and data on S3/HDFS");
System.out.println();
System.out.println("Concurrency: optimistic locking");
System.out.println(" Race -> one wins, loser retries.");
}
}
ACID properties in Delta Lake:
Atomicity: commit is one file creation
Consistency: schema enforced per write
Isolation: readers see committed versions
Durability: log and data on S3/HDFS
Concurrency: optimistic locking
Race -> one wins, loser retries.
How does schema enforcement and evolution work?
Schema enforcement rejects writes whose columns or types do not match the table, protecting against silent data corruption. Schema evolution is opt-in with .option("mergeSchema", "true"), which safely adds new columns.
# Schema enforcement: rejects mismatched write
df_bad = spark.createDataFrame(
[("T1", "extra_col_value")], ["txn_id", "unknown_col"])
try:
df_bad.write.format("delta").mode("append").save(
"s3://bank-lake/curated/txns_delta")
except Exception as e:
print("Rejected:", e)
# AnalysisException: schema mismatch
# Schema evolution: add new column safely
df_new.write.format("delta") \
.mode("append") \
.option("mergeSchema", "true") \
.save("s3://bank-lake/curated/txns_delta")
How does time travel work?
Time travel lets you query the table as it was at any prior version or timestamp. Old files remain after updates until VACUUM removes them.
# Time travel by version
df_v3 = spark.read.format("delta") \
.option("versionAsOf", 3) \
.load("s3://bank-lake/curated/txns_delta")
# Time travel by timestamp
df_yesterday = spark.read.format("delta") \
.option("timestampAsOf", "2024-10-28") \
.load("s3://bank-lake/curated/txns_delta")
# Audit: what did the table look like before the correction?
df_yesterday.show(5)
package com.bank.formats;
public class TimeTravel {
public static void main(String[] args) {
System.out.println("Time travel use cases:");
System.out.println(" 1. Audit: table before a correction job");
System.out.println(" 2. Debug: compare before/after a bug");
System.out.println(" 3. ML: reproducible training snapshots");
System.out.println(" 4. Compliance: prove historical state");
}
}
Time travel use cases:
1. Audit: table before a correction job
2. Debug: compare before/after a bug
3. ML: reproducible training snapshots
4. Compliance: prove historical state
How do UPDATE, DELETE, and MERGE work?
Delta does not mutate Parquet files in place. UPDATE and DELETE rewrite only the files containing matching rows and record the swap in the log. MERGE combines insert and update in one atomic operation.
from delta.tables import DeltaTable
delta = DeltaTable.forPath(spark,
"s3://bank-lake/curated/txns_delta")
# DELETE cancelled transactions
delta.delete("status = 'CANCELLED'")
# UPDATE a correction
delta.update(
condition="txn_id = 'T10045'",
set={"amount": "16000.00"})
# MERGE: upsert new and updated records
delta.alias("t").merge(
updates.alias("u"), "t.txn_id = u.txn_id"
).whenMatchedUpdateAll() \
.whenNotMatchedInsertAll() \
.execute()
What is VACUUM?
VACUUM removes old files that are no longer referenced by any recent version, freeing storage. You must set a retention period; the default is 7 days to allow time-travel queries.
# VACUUM: remove files older than 7 days
from delta.tables import DeltaTable
delta = DeltaTable.forPath(spark,
"s3://bank-lake/curated/txns_delta")
delta.vacuum(168) # 168 hours = 7 days
package com.bank.formats;
public class VacuumExplained {
public static void main(String[] args) {
System.out.println("VACUUM removes old unreferenced files.");
System.out.println(" Retention: 7 days (default)");
System.out.println(" After vacuum: time travel limited"
+ " to retention window.");
System.out.println(" Run periodically to control storage cost.");
}
}
VACUUM removes old unreferenced files.
Retention: 7 days (default)
After vacuum: time travel limited to retention window.
Run periodically to control storage cost.
What are common interview follow-up questions?
- How does Delta Lake compare to Apache Iceberg and Hudi?
- What is optimistic concurrency control and when does it conflict?
- How does MERGE rewrite only the affected files?
- What is the role of checkpoint files in the transaction log?
- How do you handle Delta Lake compaction (OPTIMIZE)?
Visual Overview
Plain Parquet vs Delta Lake
───────────────────────────
Plain Parquet (no coordinator) Delta Lake (transaction log)
┌──────────────────────┐ ┌──────────────────────────┐
│ part-00000.parquet │ │ _delta_log/ │
│ part-00001.parquet │ │ 0000.json (v0: add) │
│ part-00002.parquet │ │ 0001.json (v1: add) │
│ │ │ 0002.json (v2: merge) │
│ No ACID │ │ │
│ No history │ │ part-00000.parquet │
│ No schema check │ │ part-00001.parquet │
│ Concurrent read/write│ │ part-00002.parquet │
│ = partial result │ │ │
└──────────────────────┘ │ ACID, time travel, │
│ schema enforcement │
└──────────────────────────┘
MERGE Operation
───────────────
Before: After:
┌──────────┐ ┌──────────┐
│ part-001 │ │ part-001 │ (unchanged: no match)
│ part-002 │ ─rewrite─▶│ part-004 │ (rewritten: matches updated)
│ part-003 │ │ part-003 │ (unchanged)
└──────────┘ │ part-005 │ (new: inserts)
└──────────┘
Log: remove part-002, add part-004, add part-005
Key Takeaway
Delta Lake is an open-source storage layer that sits on top of Parquet files and adds a transaction log to turn a plain folder of files into a real table with ACID guarantees. Plain Parquet is only a file format with no coordinator, so concurrent writes can be seen half-finished, there is no atomic replace, and there is no history. Delta solves all three because every change is committed as one atomic log entry, and readers reconstruct a consistent snapshot by replaying the log.
Delta gives four things Parquet cannot. ACID transactions through optimistic concurrency control. Schema enforcement that rejects mismatched writes, with evolution available via mergeSchema. Time travel to query any prior version. And row-level DML: UPDATE, DELETE, and MERGE that rewrite only affected files and record the swap in the log. VACUUM removes old files after a retention period to control storage.
In a banking context at the National Bank of India, the nightly job appended transactions to a Delta table while analysts ran reports, and because readers only see committed versions, a report never picked up half-written files. When a compliance officer asked what a table looked like before a correction, time travel answered the question. And when a bad upstream feed tried to add an unexpected column, schema enforcement rejected it.
The way to summarize it in interviews is that Delta is Parquet plus a transaction log, and that log is the difference between a dumb pile of files and a trustworthy, versioned, updatable banking table.
Remember one sentence: Delta Lake is Parquet plus a transaction log that provides ACID, schema enforcement, time travel, and DML on a data lake.
---