Definition
- Data lake – a low-cost storage system, typically object storage like S3 or HDFS, that holds vast amounts of raw data in its original form using schema-on-read.
- Schema-on-read – structure is applied only when the data is queried, not when it is stored.
- Data warehouse – a managed analytical database like Snowflake or Redshift that stores cleaned, modelled, schema-on-write data optimized for fast SQL analytics.
- Schema-on-write – data must conform to a defined table structure before it is loaded.
- Data swamp – what a lake becomes without governance: nobody knows what is in which folder or whether it is trustworthy.
- Lakehouse – a hybrid that puts warehouse features (ACID, schema enforcement) on top of lake storage using Delta Lake, Iceberg, or Hudi.
Imagine the National Bank of India has two rooms. The data lake is a giant cheap warehouse where every raw document is dumped as it comes in: card logs, scanned cheque images, call recordings, app clickstreams, all thrown in boxes with no sorting, because storage is cheap and you might need it later. The data warehouse is the neat records room upstairs where only clean, labelled, filed statements live, arranged so a manager can instantly pull "Rajesh's spend by month". The lake keeps everything just in case; the warehouse keeps the polished subset ready to answer questions fast.
Understanding the Concepts
What is a data lake?
A data lake is built on cheap, massively scalable object storage. Its defining trait is schema-on-read: you store bytes now and decide their structure only when you read them. That makes it ideal for landing everything: raw JSON events, CSV files, Parquet, images, and logs, without an upfront modelling step. Lakes handle structured, semi-structured, and unstructured data equally.
# Reading raw transaction Parquet from the lake
df = spark.read.parquet(
"s3://national-bank-lake/raw/txns/dt=2024-10-29/")
df.show(3)
+--------+----------+--------+-------------------+
| txn_id|account_id| amount| created_at|
+--------+----------+--------+-------------------+
| T10045| SAV2001|15000.00|2024-10-29 09:15:00|
| T10046| SAV2001| 5000.00|2024-10-29 09:17:30|
| T10047| CUR3002|250000.0|2024-10-29 09:20:00|
+--------+----------+--------+-------------------+
What is a data warehouse?
A data warehouse is a managed analytical database using schema-on-write: you define tables and types up front, and data must conform before it is loaded. It is column-oriented and heavily optimized for analytical scans and aggregations. Modelling matters here: star schemas with fact and dimension tables are common. The warehouse holds clean, conformed, business-ready data, which is why analysts and BI tools point at it.
-- Warehouse query: fast aggregation on clean, modelled data
SELECT account_id,
SUM(amount) AS total_spend,
COUNT(*) AS txn_count
FROM warehouse.fact_transactions
WHERE txn_month = '2024-10'
GROUP BY account_id
ORDER BY total_spend DESC
LIMIT 10;
account_id | total_spend | txn_count
CUR3002 | 1250000.00 | 47
SAV2001 | 450000.00 | 123
SAV2005 | 320000.00 | 89
How do the lake and warehouse work together?
The practical relationship is that they complement each other. Raw data lands in the lake cheaply, then ETL or ELT cleans and models a refined subset into the warehouse for serving. Many banks organize the lake into zones: raw, cleansed, and curated, to add governance progressively.
package com.bank.pipeline;
import java.math.BigDecimal;
import java.util.List;
import java.util.stream.Collectors;
public class LakeToWarehouse {
record RawTxn(String txnId, String accountId,
BigDecimal amount) {}
public static void main(String[] args) {
// 1. Raw data in the lake (all records, even messy)
List<RawTxn> lake = List.of(
new RawTxn("T1", "SAV2001", new BigDecimal("15000")),
new RawTxn("T2", "SAV2001", null), // bad
new RawTxn("T3", "CUR3002", new BigDecimal("25000")));
// 2. ETL promotes only clean data to warehouse
List<RawTxn> forWarehouse = lake.stream()
.filter(t -> t.amount() != null)
.filter(t -> t.amount().compareTo(BigDecimal.ZERO) > 0)
.collect(Collectors.toList());
System.out.println("Lake records: " + lake.size());
System.out.println("Warehouse records: "
+ forWarehouse.size());
// Junk stays in the lake, never reaches the warehouse
}
}
Lake records: 3
Warehouse records: 2
What is a data swamp and how do you prevent it?
Without governance a data lake becomes a data swamp: nobody knows what is in which folder, whether it is trustworthy, or when it was last updated. You prevent it by organizing the lake into zones, cataloguing every dataset, enforcing naming conventions, and tracking lineage.
# Lake zone structure at National Bank of India
s3://national-bank-lake/
raw/ # Untouched source data
txns/
card_swipes/
loan_events/
cleansed/ # Validated, deduplicated
txns/
curated/ # Business-ready, modelled
daily_summaries/
customer_360/
package com.bank.pipeline;
public class LakeGovernance {
public static void main(String[] args) {
String[] zones = {"raw", "cleansed", "curated"};
for (String zone : zones) {
System.out.println("Zone: " + zone
+ " | Catalogued: yes"
+ " | Owner: data-eng-team"
+ " | Retention: 7 years");
}
}
}
Zone: raw | Catalogued: yes | Owner: data-eng-team | Retention: 7 years
Zone: cleansed | Catalogued: yes | Owner: data-eng-team | Retention: 7 years
Zone: curated | Catalogued: yes | Owner: data-eng-team | Retention: 7 years
How is a data lake different from a data warehouse?
| Aspect | Data Lake | Data Warehouse |
|---|---|---|
| Storage | Object storage (S3, HDFS) | Managed DB (Snowflake, Redshift) |
| Schema | Schema-on-read | Schema-on-write |
| Data types | Structured + semi + unstructured | Structured only |
| Cost | Pennies per GB | Dollars per GB |
| Query speed | Slower (no optimizer) | Fast (column-oriented, optimizer) |
| Governance | Manual (catalog, zones) | Built-in (access control, types) |
| Best for | Raw ingestion, ML features | BI reporting, regulated analytics |
# Lake: schema applied at read time
df_lake = spark.read.schema(my_schema).json(
"s3://national-bank-lake/raw/events/")
# Warehouse: schema enforced at write time
# INSERT fails if columns or types do not match
What is a lakehouse?
A lakehouse adds warehouse-like features directly on top of lake storage using table formats like Delta Lake, Apache Iceberg, or Hudi. It provides ACID transactions, schema enforcement and evolution, time travel, and row-level updates on Parquet files in the lake. The goal is to combine the lake's cheap flexibility with the warehouse's reliability in one platform.
# Lakehouse: Delta Lake on S3 with ACID and time travel
df.write.format("delta").mode("append").save(
"s3://national-bank-lake/curated/txns_delta")
# Time travel: read the table as it was yesterday
spark.read.format("delta") \
.option("timestampAsOf", "2024-10-28") \
.load("s3://national-bank-lake/curated/txns_delta") \
.show()
When should you use a lake vs a warehouse vs a lakehouse?
Use a pure lake when you want the cheapest possible storage for vast raw or unstructured data, machine learning inputs, or landing data whose shape you have not decided, and you do not need transactions. Use a warehouse when you need fast, highly concurrent, governed SQL on clean structured data for business intelligence and regulated reporting. Use a lakehouse when you want to avoid maintaining two systems and need both cheap storage and reliable, updatable analytics in one place.
package com.bank.pipeline;
public class WhenToUseWhich {
public static void main(String[] args) {
System.out.println("Lake: cheap raw storage, ML inputs");
System.out.println("Warehouse: fast governed SQL, BI reports");
System.out.println("Lakehouse: one platform, ACID on lake");
System.out.println();
System.out.println("Many banks run a hybrid:");
System.out.println(" raw -> lake");
System.out.println(" curated -> lakehouse (Delta/Iceberg)");
System.out.println(" high-concurrency BI -> warehouse");
}
}
Lake: cheap raw storage, ML inputs
Warehouse: fast governed SQL, BI reports
Lakehouse: one platform, ACID on lake
Many banks run a hybrid:
raw -> lake
curated -> lakehouse (Delta/Iceberg)
high-concurrency BI -> warehouse
What should a senior engineer know?
A senior engineer understands the cost and governance trade-offs. The lake is pennies per gigabyte but slower and ungoverned without effort. The warehouse costs more but delivers governed, interactive SQL. The lakehouse is promising but sometimes less mature for the most demanding low-latency BI concurrency. Many banks run a hybrid: raw in the lake, curated in a lakehouse table format, and a dedicated warehouse for the most latency-sensitive reports.
# Senior pattern: lake zones + lakehouse + warehouse
# Raw zone (lake): everything, cheap
raw = spark.read.parquet("s3://bank-lake/raw/txns/")
# Curated zone (lakehouse): Delta with ACID
curated = spark.read.format("delta").load(
"s3://bank-lake/curated/txns_delta")
# Serving (warehouse): Redshift for high-concurrency BI
# COPY curated Parquet into Redshift for dashboards
What are common interview follow-up questions?
- How do you prevent a data lake from becoming a data swamp?
- What is the difference between schema-on-read and schema-on-write?
- How does Delta Lake add ACID transactions on top of Parquet?
- When would you still use a separate warehouse alongside a lakehouse?
- How do you handle sensitive data (PII) in a data lake?
Visual Overview
┌──────────────────────────────────────────────────────────┐
│ DATA LAKE vs DATA WAREHOUSE │
└──────────────────────────────────────────────────────────┘
Data Lake (S3 / HDFS) Data Warehouse (Snowflake / Redshift)
───────────────────── ──────────────────────────────────────
┌────────────────────┐ ┌────────────────────────┐
│ Schema-on-READ │ │ Schema-on-WRITE │
│ │ │ │
│ raw JSON, CSV, │ ETL/ELT │ Clean, typed, modelled│
│ Parquet, images, │────────────▶│ Star schema │
│ logs, anything │ │ Fast SQL, governed │
│ │ │ │
│ Cheap (pennies/GB)│ │ Expensive (dollars/GB)│
└────────────────────┘ └────────────────────────┘
│ │
▼ ▼
ML models, raw archive BI dashboards, reports
Lakehouse (Delta Lake / Iceberg on S3) ────────────────────────────────────── ┌─────────────────────────────────────┐ │ Lake storage + Warehouse features│ │ │ │ Parquet files on S3 │ │ + _delta_log (ACID, versioning) │ │ + Schema enforcement │ │ + Time travel │ │ + UPDATE / DELETE / MERGE │ └─────────────────────────────────────┘
Key Takeaway
A data lake is a low-cost, massively scalable storage system, usually object storage like S3 or HDFS, that holds all raw data in its original form, whether structured, semi-structured, or unstructured. Its defining trait is schema-on-read, meaning you store the bytes now and apply structure only when you query them, which is why a lake is perfect for landing everything cheaply in case you need it later. A data warehouse, by contrast, is a managed analytical database like Snowflake or Redshift that uses schema-on-write, so data must be cleaned and modelled before it is loaded. It is column-oriented and tuned for fast analytical scans, so business queries run quickly.
The two are not competitors: they work together. Raw data lands in the lake cheaply, then ETL or ELT cleans and models a refined subset into the warehouse for serving, and many banks organize the lake into raw, cleansed, and curated zones to add governance and avoid turning it into a data swamp where nobody trusts anything. A senior point is the lakehouse, which uses table formats like Delta Lake, Iceberg, or Hudi to put warehouse-like ACID transactions, schema enforcement, and time travel directly on top of lake storage, so you get one platform instead of constantly copying data between two.
In a banking context at the National Bank of India, the lake kept every raw event forever and cheaply, including messy or incomplete records, because a future fraud model might need them. A curation job promoted only the valid records into the warehouse, dropping junk like a null-amount row, and the warehouse answered fast aggregate queries such as total spend per account for Rajesh and Priya. The important design idea was that junk stayed safely in the lake but was never allowed to pollute the governed warehouse.
This gives the bank both cheap completeness and trustworthy, interactive analytics in one architecture.
Remember one sentence: a data lake stores everything cheaply with schema-on-read, a data warehouse stores clean data fast with schema-on-write, and a lakehouse combines both on one platform.
---