Definition
Delta Lake time travel is the ability to query historical states of a table.
- VERSION AS OF n – Reads the table as it existed at commit version n.
- TIMESTAMP AS OF ts – Reads the table as it existed at the latest commit at or before timestamp ts.
- Immutable log – Every write produces a new version in
_delta_log, and old data files remain on disk until VACUUM removes them. - RESTORE TABLE – Rolls the table back to a prior version by creating a new commit, preserving history rather than erasing it.
Primary use cases are debugging, auditing, rollback of bad writes, and reproducible machine learning.
The National Bank of India's ledger is append-only and numbered. If someone asks "what was Rajesh's balance table at 9 AM, before the correction batch ran?", you do not need a backup: you just flip the ledger back to that page number. Delta's transaction log is exactly that numbered ledger; time travel is flipping back to any page by version number or timestamp.
Understanding the Concepts
What is Delta Lake time travel?
Every Delta write produces a new version N with its own commit file in _delta_log. The commit records which files were added and removed, but the removed files stay on disk until VACUUM deletes them after the retention window. Time travel reconstructs the file set for a chosen version by replaying the log only up to that point.
-- View the history of a Delta table
DESCRIBE HISTORY banking_prod.accounts;
version timestamp operation userName
------- --------------------- ---------- --------
5 2025-06-15 14:30:00 MERGE etl_job
4 2025-06-15 09:00:00 UPDATE correction_batch
3 2025-06-14 22:00:00 MERGE nightly_load
2 2025-06-14 09:00:00 UPDATE manual_fix
1 2025-06-13 22:00:00 MERGE nightly_load
0 2025-06-13 10:00:00 CREATE setup_job
How do you read a past version?
In SQL you use VERSION AS OF or TIMESTAMP AS OF. In the DataFrame API you use the versionAsOf and timestampAsOf read options.
-- Read version 3 (before the correction batch)
SELECT account_id, balance
FROM banking_prod.accounts VERSION AS OF 3
WHERE account_id = 'ACC-1001';
-- Read by timestamp (latest commit at or before 9 AM)
SELECT account_id, balance
FROM banking_prod.accounts TIMESTAMP AS OF '2025-06-15 09:00:00'
WHERE account_id = 'ACC-1001';
# DataFrame API
version_3 = spark.read.format("delta") \
.option("versionAsOf", 3) \
.load("/data/bank/accounts")
timestamp_read = spark.read.format("delta") \
.option("timestampAsOf", "2025-06-15 09:00:00") \
.load("/data/bank/accounts")
# Compare before and after the correction batch
before = spark.read.format("delta").option("versionAsOf", 3).load("/data/bank/accounts")
after = spark.read.format("delta").option("versionAsOf", 4).load("/data/bank/accounts")
changes = before.join(after, "account_id").filter("before.balance != after.balance")
How do you use time travel for debugging?
Compare a known-good version with a suspect one to see exactly what a job changed. At the National Bank of India, this is how we investigate incorrect balance updates.
# Debugging: what did the correction batch change?
before_correction = spark.read.format("delta") \
.option("versionAsOf", 3).load("/data/bank/accounts") \
.select("account_id", col("balance").alias("balance_before"))
after_correction = spark.read.format("delta") \
.option("versionAsOf", 4).load("/data/bank/accounts") \
.select("account_id", col("balance").alias("balance_after"))
diff = before_correction.join(after_correction, "account_id") \
.filter("balance_before != balance_after") \
.withColumn("change", col("balance_after") - col("balance_before"))
diff.show()
account_id balance_before balance_after change
ACC-1001 50000.00 45000.00 -5000.00
ACC-2045 120000.00 125000.00 5000.00
How do you use time travel for auditing?
Regulators often demand that you reproduce the precise state a report was based on. Time travel lets you do this without maintaining separate backups.
-- Reproduce the exact data the end-of-day report used
-- The report ran at version 3, so we read that version
SELECT branch_code, SUM(balance) AS total_deposits
FROM banking_prod.accounts VERSION AS OF 3
GROUP BY branch_code;
-- Auditor can verify: this is exactly what the report computed
How do you rollback a bad write?
RESTORE TABLE creates a new commit whose data equals the old version, so you recover without destroying history. The rollback itself becomes a new version in the log.
-- The correction batch (version 4) wrote wrong values
-- Restore to version 3 (the last known-good state)
RESTORE TABLE banking_prod.accounts TO VERSION AS OF 3;
-- This creates version 6 whose data equals version 3
-- Versions 4 and 5 still exist in history for audit
DESCRIBE HISTORY banking_prod.accounts;
version operation
6 RESTORE
5 MERGE
4 UPDATE (bad correction)
3 MERGE (good nightly load)
How do you use time travel for reproducible ML?
Pin a training read to a fixed version so a model always trains on identical data, making experiments repeatable.
# Reproducible ML: pin training data to a fixed version
MODEL_TRAINING_VERSION = 3
training_data = spark.read.format("delta") \
.option("versionAsOf", MODEL_TRAINING_VERSION) \
.load("/data/bank/features/credit_risk")
# Every run of this notebook trains on identical data
# Version number is logged in MLflow for traceability
import mlflow
mlflow.log_param("data_version", MODEL_TRAINING_VERSION)
What is the retention window and VACUUM interaction?
Time travel only works within the retention window. VACUUM permanently deletes old files after the retention period (default 7 days). If you vacuum aggressively, you destroy the files that historical versions relied on.
-- Default retention: 7 days
VACUUM banking_prod.accounts;
-- Set custom retention for audit requirements
ALTER TABLE banking_prod.accounts
SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '30 days');
-- DANGER: vacuuming with 0 hours destroys all time travel
-- Delta refuses by default; you must override the safety check
-- DO NOT DO THIS in production banking tables
-- SET spark.databricks.delta.retentionDurationCheck.enabled = false;
-- VACUUM banking_prod.accounts RETAIN 0 HOURS;
What should a senior engineer know?
- For strict reproducibility, prefer version numbers over timestamps. Timestamps can be surprising around clock boundaries and daylight saving changes.
- If you must keep a version forever (regulatory requirement), clone or archive it before VACUUM removes the underlying files.
- Time travel reads reconstruct the file set, not a separate snapshot, so they are as fast as a normal read.
- Delta's Change Data Feed (CDF) feature can emit row-level changes between versions, which is useful for incremental downstream processing.
-- Clone a table at a specific version for permanent archival
CREATE TABLE banking_archive.accounts_2025q2
DEEP CLONE banking_prod.accounts VERSION AS OF 3;
-- This clone is independent and survives VACUUM on the source
Visual Overview
Delta Table Versions:
_delta_log/
├── 00000.json (CREATE) data files: [f1, f2]
├── 00001.json (MERGE) add: [f3] remove: [f1]
├── 00002.json (UPDATE) add: [f4] remove: [f2]
├── 00003.json (MERGE) add: [f5] remove: [f3]
└── 00004.json (UPDATE) add: [f6] remove: [f4]
VERSION AS OF 2:
Replay log 0→1→2 → file set = {f3, f4}
(f1 removed by v1, f2 removed by v2)
Current version (4):
File set = {f5, f6}
VACUUM after retention:
Deletes f1, f2, f3, f4 → versions 0-2 become unreadable
Key Takeaway
Delta Lake time travel is the ability to read a table exactly as it existed at a past version or point in time. In SQL you use VERSION AS OF a number or TIMESTAMP AS OF a timestamp, and in the DataFrame API you use the versionAsOf and timestampAsOf read options. It works because every write commits a new immutable version in the transaction log, and old data files stay on disk until VACUUM removes them.
The use cases are debugging (compare versions to see what a job changed), auditing (reproduce the exact state a report was based on), rollback (RESTORE TABLE creates a new commit equaling the old version), and reproducible machine learning (pin training reads to a fixed version). At the National Bank of India, this saved us when a correction batch overwrote balances with wrong values: we read the pre-batch version to confirm the damage, then ran RESTORE to roll forward to the good state while keeping every version in the audit trail.
The caution is that time travel only works within the retention window. VACUUM permanently deletes old files, so if you must keep a version forever, clone or archive it first. For strict reproducibility, prefer version numbers over timestamps.
Remember one sentence: every Delta commit is an immutable snapshot, so VERSION AS OF or TIMESTAMP AS OF lets you read, debug, audit, or rollback any state within the retention window.
---