Definition
- Databricks – a cloud-based unified data and AI platform created by the original authors of Apache Spark.
- Databricks Runtime – a performance-tuned distribution of Spark with optimizations like the Photon vectorized engine.
- Delta Lake – the default storage layer that brings ACID transactions, schema enforcement, and time travel to Parquet files.
- Unity Catalog – centralized governance providing access control, auditing, data lineage, and discovery across workspaces.
- Notebooks – collaborative, versioned cells where engineers and analysts write and run code against shared clusters.
- Lakehouse – the thesis that Delta Lake plus Spark plus SQL endpoints replaces the need for a separate lake and warehouse.
Imagine Apache Spark is a very powerful engine, like a bare jet engine sitting on a workshop floor. You could, with great effort, build your own aircraft around it, wire the controls, hire mechanics, and run the airport yourself. Databricks is like buying a complete, maintained aircraft from the people who designed that engine, with autopilot, a tuned version of the engine that runs faster, a cockpit where the crew works together, and ground staff who handle fuelling and maintenance. At the National Bank of India you still decide where to fly and what cargo to carry, but you are not building and maintaining the plane and the airport; you focus on the journey.
Understanding the Concepts
What is Apache Spark and why does it need Databricks?
Apache Spark is the open-source distributed compute engine, but using it in production means provisioning machines, installing Spark, configuring the cluster manager, tuning memory, setting up storage, managing libraries, securing it, and monitoring it. Databricks was founded by the same people who created Spark to deliver all of that as a managed cloud service.
# Same PySpark code runs on vanilla Spark or Databricks
df = spark.read.parquet(
"s3://national-bank-lake/raw/txns/")
result = df.groupBy("account_id").sum("amount")
result.show(5)
+----------+-----------+
|account_id|sum(amount)|
+----------+-----------+
| SAV2001| 450000.00|
| CUR3002| 1250000.00|
| SAV2005| 320000.00|
+----------+-----------+
# Same code, but Databricks Runtime runs it faster
What is the Databricks Runtime?
The Databricks Runtime is a performance-tuned distribution of Spark with extra optimizations such as the Photon vectorized engine, caching improvements, and tuned defaults. The same code often runs considerably faster than on vanilla Spark without any changes.
package com.bank.platform;
public class DatabricksRuntime {
public static void main(String[] args) {
System.out.println("Databricks Runtime optimizations:");
System.out.println(" 1. Photon: vectorized C++ engine");
System.out.println(" 2. Adaptive query execution tuned");
System.out.println(" 3. Caching improvements");
System.out.println(" 4. Optimized shuffle service");
System.out.println();
System.out.println("Same PySpark/Scala/SQL code,"
+ " faster execution.");
}
}
Databricks Runtime optimizations:
1. Photon: vectorized C++ engine
2. Adaptive query execution tuned
3. Caching improvements
4. Optimized shuffle service
Same PySpark/Scala/SQL code, faster execution.
What is Delta Lake and why is it the default?
Delta Lake, created by Databricks and open-sourced, is the default storage layer. It brings ACID transactions, schema enforcement, time travel, and efficient upserts on top of Parquet files in cloud object storage, turning a raw data lake into a reliable lakehouse.
# Delta Lake: ACID on the lake
df.write.format("delta").mode("append").save(
"s3://national-bank-lake/curated/txns_delta")
# Time travel: read yesterday's version
spark.read.format("delta") \
.option("timestampAsOf", "2024-10-28") \
.load("s3://national-bank-lake/curated/txns_delta") \
.show(3)
# MERGE: upsert atomically
from delta.tables import DeltaTable
delta_table = DeltaTable.forPath(spark,
"s3://national-bank-lake/curated/txns_delta")
delta_table.alias("t").merge(
updates.alias("u"), "t.txn_id = u.txn_id"
).whenMatchedUpdateAll() \
.whenNotMatchedInsertAll() \
.execute()
What is Unity Catalog?
Unity Catalog provides centralized governance: one place to manage access control, auditing, data lineage, and discovery across workspaces. This matters enormously in a regulated industry like banking.
-- Unity Catalog: fine-grained access control
GRANT SELECT ON TABLE curated.transactions
TO `analysts@nationalbank.in`;
-- Column-level masking for PII
ALTER TABLE curated.transactions
ALTER COLUMN pan_number
SET MASK mask_pan;
-- Data lineage: trace where data came from
-- Unity Catalog automatically tracks which tables
-- feed into which downstream tables
package com.bank.platform;
public class UnityCatalogBenefits {
public static void main(String[] args) {
System.out.println("Unity Catalog provides:");
System.out.println(" 1. Access control (table, column level)");
System.out.println(" 2. Audit logging (who read what, when)");
System.out.println(" 3. Data lineage (source -> target)");
System.out.println(" 4. Discovery (search across all data)");
System.out.println();
System.out.println("Critical for banking regulation.");
}
}
Unity Catalog provides:
1. Access control (table, column level)
2. Audit logging (who read what, when)
3. Data lineage (source -> target)
4. Discovery (search across all data)
Critical for banking regulation.
How does cluster management work?
Cluster management is automated with autoscaling, auto-termination of idle clusters, and different cluster types for interactive versus production work. This controls cost and reduces operational burden.
{
"cluster_name": "bank-etl-prod",
"spark_version": "14.3.x-scala2.12",
"autoscale": {
"min_workers": 2,
"max_workers": 20
},
"autotermination_minutes": 30,
"node_type_id": "i3.xlarge",
"driver_node_type_id": "i3.xlarge"
}
package com.bank.platform;
public class ClusterManagement {
public static void main(String[] args) {
System.out.println("Cluster types:");
System.out.println(" Interactive: notebooks, ad-hoc analysis");
System.out.println(" Job: scheduled ETL, auto-terminate after");
System.out.println();
System.out.println("Cost controls:");
System.out.println(" Autoscale: 2 to 20 workers based on load");
System.out.println(" Auto-terminate: shut down after 30 min idle");
System.out.println(" Spot instances: 60-90% cheaper for fault-tolerant jobs");
}
}
Cluster types:
Interactive: notebooks, ad-hoc analysis
Job: scheduled ETL, auto-terminate after
Cost controls:
Autoscale: 2 to 20 workers based on load
Auto-terminate: shut down after 30 min idle
Spot instances: 60-90% cheaper for fault-tolerant jobs
What is the lakehouse thesis?
The lakehouse thesis is that you no longer need a separate data lake and warehouse. Delta Lake plus Spark plus SQL endpoints give you cheap storage and warehouse-style performance and reliability in one place.
# SQL endpoint: warehouse-like experience on Delta Lake
# Analysts query with plain SQL, no Spark knowledge needed
# SELECT * FROM curated.daily_summary
# WHERE branch = 'Mumbai Central'
# AND month = '2024-10'
# Under the hood: Photon engine on Delta Parquet files
# Same data, lake storage cost, warehouse query speed
package com.bank.platform;
public class LakehouseThesis {
public static void main(String[] args) {
System.out.println("Before lakehouse:");
System.out.println(" Lake (raw) -> ETL -> Warehouse (clean)");
System.out.println(" Two systems, data copied between them.");
System.out.println();
System.out.println("With lakehouse:");
System.out.println(" Lake storage + Delta log = one system");
System.out.println(" ETL, analytics, ML on same data.");
System.out.println(" No copying between lake and warehouse.");
}
}
Before lakehouse:
Lake (raw) -> ETL -> Warehouse (clean)
Two systems, data copied between them.
With lakehouse:
Lake storage + Delta log = one system
ETL, analytics, ML on same data.
No copying between lake and warehouse.
What are the trade-offs of adopting Databricks?
Databricks is not a fork of Spark that locks you in at the code level. Your Spark code is portable and Delta Lake is open source. However, features like Photon, Unity Catalog, Delta Live Tables, and managed services are Databricks-specific value you give up if you move off the platform.
package com.bank.platform;
public class DatabricksTradeoffs {
public static void main(String[] args) {
System.out.println("Portable (open source):");
System.out.println(" - PySpark / Scala / SQL code");
System.out.println(" - Delta Lake format and protocol");
System.out.println(" - MLflow experiment tracking");
System.out.println();
System.out.println("Databricks-specific (not portable):");
System.out.println(" - Photon engine");
System.out.println(" - Unity Catalog");
System.out.println(" - Delta Live Tables");
System.out.println(" - Managed clusters and notebooks");
System.out.println();
System.out.println("Trade: cost + some lock-in for"
+ " less ops + faster execution + governance.");
}
}
Portable (open source):
- PySpark / Scala / SQL code
- Delta Lake format and protocol
- MLflow experiment tracking
Databricks-specific (not portable):
- Photon engine
- Unity Catalog
- Delta Live Tables
- Managed clusters and notebooks
Trade: cost + some lock-in for less ops + faster execution + governance.
What are common interview follow-up questions?
- How does the Photon engine differ from the standard Spark engine?
- What is Delta Live Tables and how does it simplify ETL?
- How does Unity Catalog compare to AWS Glue Catalog or Hive Metastore?
- What is the difference between a Databricks job cluster and an all-purpose cluster?
- How do you estimate Databricks cost for a given workload?
Visual Overview
Databricks Platform Architecture ──────────────────────────────── ┌──────────────────────────────────────────────┐ │ DATABRICKS PLATFORM │ │ │ │ ┌─────────────┐ ┌───────────┐ ┌─────────┐│ │ │ Notebooks │ │ Jobs │ │ SQL ││ │ │(interactive)│ │(scheduled)│ │Endpoints││ │ └──────┬──────┘ └─────┬─────┘ └────┬────┘│ │ │ │ │ │ │ └───────┬───────┘──────────────┘ │ │ ▼ │ │ ┌──────────────────────────────────┐ │ │ │ Databricks Runtime │ │ │ │ (Optimized Spark + Photon) │ │ │ └──────────────┬───────────────────┘ │ │ │ │ │ ┌──────────────▼───────────────────┐ │ │ │ Delta Lake (ACID) │ │ │ │ Parquet + _delta_log on S3 │ │ │ └──────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────┐ │ │ │ Unity Catalog │ │ │ │ Access control, lineage, audit │ │ │ └──────────────────────────────────┘ │ └──────────────────────────────────────────────┘
Key Takeaway
Databricks is a cloud-based unified data and AI platform built by the original creators of Apache Spark. The simplest way to describe the relationship is that Databricks is the managed, optimized, enterprise environment around open-source Spark. You still write Spark code in PySpark, SQL, Scala, or R, but it runs on the Databricks Runtime, a performance-tuned distribution with Photon and other optimizations, and you are freed from provisioning, patching, and babysitting clusters.
Around the engine, Databricks adds the pieces a real data platform needs. Delta Lake is the default storage layer, bringing ACID transactions, schema enforcement, time travel, and efficient upserts on top of Parquet. Unity Catalog provides centralized governance for access control, auditing, and lineage. Cluster management is automated with autoscaling and auto-termination. Collaborative notebooks let teams work together, and MLflow integrates machine learning end to end.
The headline positioning is the lakehouse thesis: Delta Lake plus Spark plus SQL endpoints replaces the need for a separate lake and warehouse, giving cheap storage and warehouse-grade reliability in one place. The honest trade-off is cost and some platform dependence, because while Spark code and Delta are open and portable, features like Photon and Unity Catalog are Databricks-specific.
In a banking context at the National Bank of India, moving off self-managed Spark meant the same jobs ran faster on the tuned runtime, clusters autoscaled and auto-terminated to save money, Unity Catalog gave auditors the governance they required, and Delta Lake collapsed the brittle lake-plus-warehouse split into one lakehouse with real transactions and time travel.
Remember one sentence: Databricks is Spark's managed, optimized platform with Delta Lake, governance, and notebooks, so you focus on data instead of infrastructure.
---