Definition
- Data pipeline – an orchestrated series of connected steps that move data from one or more sources to one or more destinations, applying transformations along the way.
- Stage – a single-responsibility unit in the pipeline: ingestion, validation, enrichment, or loading.
- DAG – a directed acyclic graph that models stage dependencies so downstream stages only run after upstream stages succeed.
- Idempotency – the property that re-running a stage produces the same result, achieved through upserts keyed by a stable id.
- Dead-letter queue (DLQ) – a quarantine for poison records that keep failing, so one bad message never blocks the flow.
- Observability – per-stage row counts, latency, error rates, and lag that let operators see the pipeline's health.
Think of the National Bank of India's cheque-clearing line. A cheque enters at one end, then moves through stations: a scanner reads it, a checker removes damaged ones, a clerk matches it to the right account, and finally it is filed in the ledger. A data pipeline is exactly that assembly line, but for data: an event enters from Kafka, a cleaning stage drops bad records, a join stage attaches Rajesh's customer details, and a load stage files it into the warehouse. Each station does one job and hands its output to the next.
Understanding the Concepts
What is a data pipeline?
A data pipeline is an orchestrated series of connected steps that move data from one or more sources to one or more destinations, applying transformations along the way. Each stage consumes the output of the previous stage, and the whole flow is scheduled, monitored, retried on failure, and designed to be reliable, idempotent, and observable.
package com.bank.pipeline;
public class PipelineOverview {
public static void main(String[] args) {
String[] stages = {
"1. Ingest from Kafka",
"2. Validate and clean",
"3. Join with customer table",
"4. Upsert into warehouse"
};
for (String stage : stages) {
System.out.println(stage);
}
}
}
1. Ingest from Kafka
2. Validate and clean
3. Join with customer table
4. Upsert into warehouse
How is a pipeline modelled as a DAG?
A pipeline is modelled as a directed acyclic graph (DAG) where each node is a stage and each edge is a dependency. This makes dependencies explicit: the load stage only runs after the join stage succeeds, and independent branches can run in parallel.
# Airflow DAG for National Bank transaction pipeline
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
with DAG("bank_txn_pipeline",
start_date=datetime(2024, 1, 1),
schedule_interval="0 2 * * *",
catchup=False) as dag:
ingest = PythonOperator(
task_id="ingest_from_kafka",
python_callable=read_kafka_topic)
clean = PythonOperator(
task_id="validate_and_clean",
python_callable=filter_bad_records)
enrich = PythonOperator(
task_id="join_customer_table",
python_callable=enrich_with_customer)
load = PythonOperator(
task_id="upsert_warehouse",
python_callable=upsert_to_redshift,
retries=3)
ingest >> clean >> enrich >> load
What makes a pipeline idempotent?
Idempotency means re-running a stage produces the same result. You achieve it with upserts keyed by a stable id so a retry never duplicates data. This is critical in banking because a duplicated transaction in a report can mislead a regulator about real money.
package com.bank.pipeline;
import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
public class IdempotentPipeline {
static final String UPSERT = """
INSERT INTO warehouse.transactions
(txn_id, account_id, amount)
VALUES (?, ?, ?)
ON CONFLICT (txn_id)
DO UPDATE SET amount = EXCLUDED.amount
""";
public static void main(String[] args) throws Exception {
try (Connection conn = DriverManager.getConnection(
"jdbc:postgresql://localhost/warehouse",
"pipeline", "secret");
PreparedStatement ps = conn.prepareStatement(UPSERT)) {
// First run: inserts
ps.setString(1, "T10045");
ps.setString(2, "SAV2001");
ps.setBigDecimal(3, new BigDecimal("15000.00"));
ps.executeUpdate();
// Second run (retry): updates, no duplicate
ps.setString(1, "T10045");
ps.setString(2, "SAV2001");
ps.setBigDecimal(3, new BigDecimal("15000.00"));
ps.executeUpdate();
System.out.println("Two runs, one row. Idempotent.");
}
}
}
Two runs, one row. Idempotent.
What is a dead-letter queue?
A dead-letter queue (DLQ) captures poison records that keep failing so one bad message never blocks the whole pipeline. The DLQ is owned and alerted, not silently dropped. Engineers investigate DLQ records to fix root causes.
package com.bank.pipeline;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
public class DeadLetterQueue {
record Transaction(String txnId, BigDecimal amount) {}
public static void main(String[] args) {
List<Transaction> incoming = List.of(
new Transaction("T1", new BigDecimal("15000")),
new Transaction("T2", null), // poison
new Transaction("T3", new BigDecimal("25000")));
List<Transaction> good = new ArrayList<>();
List<Transaction> dlq = new ArrayList<>();
for (Transaction t : incoming) {
if (t.amount() == null) {
dlq.add(t);
System.out.println("DLQ: " + t.txnId()
+ " (null amount)");
} else {
good.add(t);
}
}
System.out.println("Processed: " + good.size());
System.out.println("Dead-letter: " + dlq.size());
}
}
DLQ: T2 (null amount)
Processed: 2
Dead-letter: 1
What is the difference between batch and streaming pipelines?
Batch pipelines run on a schedule over a bounded chunk of data. Streaming pipelines run continuously over an unbounded event stream. Many real systems combine both: streaming for low-latency alerts and batch for complete daily reports.
package com.bank.pipeline;
public class BatchVsStream {
public static void main(String[] args) {
System.out.println("Batch pipeline:");
System.out.println(" Schedule: nightly at 2 AM");
System.out.println(" Input: all txns for 2024-10-29");
System.out.println(" Output: daily summary in warehouse");
System.out.println();
System.out.println("Streaming pipeline:");
System.out.println(" Schedule: continuous");
System.out.println(" Input: each Kafka event as it arrives");
System.out.println(" Output: real-time fraud alert");
}
}
Batch pipeline:
Schedule: nightly at 2 AM
Input: all txns for 2024-10-29
Output: daily summary in warehouse
Streaming pipeline:
Schedule: continuous
Input: each Kafka event as it arrives
Output: real-time fraud alert
How do you make a pipeline observable?
Observability is non-negotiable for production pipelines. You track per-stage row counts, latency, error rates, and lag, and emit metrics and logs so operators can see the pipeline's health at a glance.
package com.bank.pipeline;
public class PipelineMetrics {
public static void main(String[] args) {
// Simulated per-stage metrics
int ingested = 10000;
int cleaned = 9850;
int enriched = 9850;
int loaded = 9850;
int dlq = 150;
long latencyMs = 4500;
System.out.println("Pipeline Run Metrics");
System.out.println(" Ingested: " + ingested);
System.out.println(" Cleaned: " + cleaned);
System.out.println(" Enriched: " + enriched);
System.out.println(" Loaded: " + loaded);
System.out.println(" Dead-letter: " + dlq);
System.out.println(" Latency: " + latencyMs + " ms");
if (dlq > ingested * 0.05) {
System.out.println(" ALERT: DLQ rate > 5%!");
}
}
}
Pipeline Run Metrics
Ingested: 10000
Cleaned: 9850
Enriched: 9850
Loaded: 9850
Dead-letter: 150
Latency: 4500 ms
What are data-quality gates in a pipeline?
Data-quality gates sit between stages and check invariants before data flows to the next step. If a gate fails, the batch is quarantined rather than silently poisoning the warehouse.
package com.bank.pipeline;
public class QualityGate {
public static void main(String[] args) {
int extractedRows = 10000;
int transformedRows = 9500;
double dropRate = 1.0 - ((double) transformedRows
/ extractedRows);
System.out.println("Extracted: " + extractedRows);
System.out.println("Transformed: " + transformedRows);
System.out.printf("Drop rate: %.1f%%\n",
dropRate * 100);
if (dropRate > 0.10) {
System.out.println("GATE FAILED: drop rate > 10%."
+ " Quarantining batch.");
} else {
System.out.println("GATE PASSED: proceeding to load.");
}
}
}
Extracted: 10000
Transformed: 9500
Drop rate: 5.0%
GATE PASSED: proceeding to load.
What are common interview follow-up questions?
- How do you handle a pipeline stage that fails midway through?
- What is backpressure and how do you handle it?
- How do you backfill historical data safely?
- What is the difference between a pipeline and a workflow?
- How do you test a data pipeline before deploying to production?
Visual Overview
Data Pipeline (Banking Transaction Flow)
─────────────────────────────────────────
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ INGEST │──▶│ VALIDATE │──▶│ ENRICH │──▶│ LOAD │
│ │ │ & CLEAN │ │ │ │ │
│ Kafka │ │ Drop null│ │ Join to │ │ Upsert │
│ events │ │ amounts │ │ customer │ │ warehouse│
└──────────┘ └────┬─────┘ └──────────┘ └──────────┘
│
▼
┌──────────┐
│ DLQ │
│ (poison │
│ records) │
└──────────┘
DAG Dependencies
────────────────
ingest ──▶ validate ──▶ enrich ──▶ load
│ │ │ │
▼ ▼ ▼ ▼
[ckpt] [gate] [gate] [retry x3]
Parallel branches:
ingest_cards ──▶ clean_cards ──┐
├──▶ merge ──▶ load
ingest_loans ──▶ clean_loans ──┘
Key Takeaway
A data pipeline is an orchestrated series of connected steps that move data from one or more sources to one or more destinations, transforming it along the way. Each stage consumes the output of the previous stage, and a common banking example is read from Kafka, clean bad records, join with the customer table, and write to the warehouse. Modelling the flow as a directed acyclic graph makes dependencies explicit, so the load stage only runs after the join succeeds, and independent branches can run in parallel.
What turns a pile of scripts into a real pipeline is orchestration and reliability. Orchestration with a tool like Airflow defines the DAG, schedules runs, enforces dependencies, retries failed tasks, and alerts on failure. On the reliability side, design for idempotency using upserts keyed by a stable id so a retry never duplicates data, use checkpoints and watermarks so a pipeline resumes rather than reprocessing everything, handle backpressure so a fast producer does not overwhelm a slow stage, and route poison records to a dead-letter queue so one bad message never blocks the whole flow.
Data-quality gates between stages quarantine bad extracts rather than silently poisoning the warehouse, and observability with per-stage row counts, latency, error rates, and lag makes the pipeline's health visible at a glance.
In a banking context at the National Bank of India, the transaction pipeline ingested raw events from Kafka, cleaned them while diverting poison records to a dead-letter queue, joined each clean event to the customer table to attach Rajesh's or Priya's name, and upserted the enriched records into the warehouse keyed by transaction id. Because the load was idempotent, rerunning the pipeline never duplicated data, and because bad records went to the DLQ, a single poison message never brought the run down.
Remember one sentence: a data pipeline is an orchestrated, idempotent assembly line that moves data from source to destination, transforming it reliably at each stage.
---