What is data engineering | Data Engineering

Definition

Data engineering builds the reliable plumbing that turns messy raw data into clean, query-ready data for everyone else.

Data engineering – the discipline of designing, building, and operating the systems and pipelines that collect raw data from many sources, move it reliably, clean and transform it, and store it in a shape that downstream consumers can query quickly.
Pipeline – an orchestrated series of stages (ingest, validate, transform, load) that moves data from source to destination.
Ingestion – extracting data from transactional databases, event streams, APIs, or files without disturbing the source.
Transformation – cleaning nulls, deduplicating, joining to reference data, converting types, and computing derived fields.
Orchestration – scheduling and sequencing pipeline stages with dependency tracking and automatic retries using tools like Apache Airflow.
Data quality – the reliability layer that makes a pipeline production-grade: idempotency, schema evolution, late-data handling, and monitoring.

Imagine the National Bank of India generates a flood of raw events all day: Rajesh swipes his card, Priya transfers rent, ATMs dispense cash, loans get approved. That raw data is messy and scattered across dozens of systems. A data engineer is like the person who builds the pipes, filters, and warehouses of the bank's data plumbing: they collect every drop of raw data, clean out the dirt, and store it neatly in labelled tanks so that the analytics team can instantly answer "how much did Rajesh spend last month" without digging through the raw sewage themselves.

Understanding the Concepts

What is data engineering?

Data engineering is the discipline of building and operating systems that collect raw data from many sources, move it reliably, clean and transform it, and store it in a shape that downstream consumers like analysts, data scientists, dashboards, and machine learning models can query quickly and trust. A data engineer owns the full pipeline: ingestion, storage, transformation, orchestration, data quality, and the reliability and cost of all of it at scale.

In a banking context, data engineering sits upstream of every report, every fraud model, and every customer dashboard. Without it, raw transaction events stay scattered across dozens of source systems and nobody can answer a simple question like "what was Rajesh's total card spend last month" without manually digging through logs.

JAVA

package com.bank.pipeline;

import java.math.BigDecimal;
import java.util.List;
import java.util.stream.Collectors;

public class DataEngineeringOverview {
    public static void main(String[] args) {
        // A data engineer builds the plumbing that turns raw events
        // into clean, query-ready records for everyone else
        List<String> responsibilities = List.of(
            "Ingestion",
            "Storage",
            "Transformation",
            "Orchestration",
            "Data Quality"
        );
        responsibilities.forEach(r ->
            System.out.println("Responsibility: " + r));
    }
}

TEXT

Responsibility: Ingestion
Responsibility: Storage
Responsibility: Transformation
Responsibility: Orchestration
Responsibility: Data Quality

What are the core responsibilities of a data engineer?

The work breaks into five repeating responsibilities that apply to every project.

Ingestion gets data out of source systems: transactional databases via JDBC or change-data-capture, Kafka event streams, external APIs like a credit bureau, and flat files like end-of-day settlement CSVs.
Storage decides where data lands: a cheap data lake for raw data, a column-oriented warehouse for fast analytics, or both.
Transformation cleans, standardizes, joins, deduplicates, and aggregates: the T in ETL.
Orchestration schedules and sequences stages so downstream jobs only run after upstream jobs succeed.
Data quality makes the pipeline production-grade: idempotency, schema evolution, monitoring, and data contracts.

JAVA

package com.bank.pipeline;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class IngestionExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "kafka-broker:9092");
        props.put("group.id", "bank-txn-ingest");
        props.put("key.deserializer",
            "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer",
            "org.apache.kafka.common.serialization.StringDeserializer");

        try (KafkaConsumer<String, String> consumer =
                 new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("bank-transactions"));
            ConsumerRecords<String, String> records =
                consumer.poll(Duration.ofMillis(500));
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("Ingested txn: " + record.value());
            }
        }
    }
}

How does ingestion work?

Ingestion is extracting data from source systems without disturbing them. Sources vary widely: a production PostgreSQL database holding accounts, a Kafka topic carrying card-swipe events, an external API from a credit bureau like CIBIL, or a nightly settlement CSV from a stock exchange.

A key senior concern is incremental extraction rather than re-reading everything each run. You only pull transactions newer than the last watermark, which keeps the extract cheap and avoids re-processing old rows.

SQL

-- Incremental extract using a watermark
SELECT txn_id, account_id, amount, created_at
FROM   transactions
WHERE  created_at > :last_loaded_timestamp
ORDER  BY created_at;

TEXT

txn_id   | account_id | amount    | created_at
T10045   | SAV2001    | 15000.00  | 2024-10-29 09:15:00
T10046   | SAV2001    | -5000.00  | 2024-10-29 09:17:30
T10047   | CUR3002    | 250000.00 | 2024-10-29 09:20:00

How does transformation work?

Transformation is where most of the real logic lives. You clean nulls and bad values, filter out irrelevant rows, deduplicate, join to reference data to attach the account holder's name to each transaction, convert types and formats, standardize currency and time zones, and compute derived fields and aggregates.

All money math must use BigDecimal because floating-point arithmetic would corrupt financial totals.

JAVA

package com.bank.pipeline;

import java.math.BigDecimal;
import java.util.List;
import java.util.stream.Collectors;

public class TransformExample {
    record Transaction(String txnId, String accountId,
                       BigDecimal amount, String currency) {}

    public static void main(String[] args) {
        List<Transaction> raw = List.of(
            new Transaction("T1", "SAV2001",
                new BigDecimal("15000.00"), "INR"),
            new Transaction("T2", "SAV2001",
                null, "INR"),           // bad: null amount
            new Transaction("T3", "CUR3002",
                new BigDecimal("-500.00"), "INR") // bad: negative
        );

        List<Transaction> clean = raw.stream()
            .filter(t -> t.amount() != null)
            .filter(t -> t.amount().compareTo(BigDecimal.ZERO) > 0)
            .collect(Collectors.toList());

        clean.forEach(t -> System.out.println(
            t.txnId() + " | " + t.accountId()
            + " | " + t.amount()));
    }
}

TEXT

T1 | SAV2001 | 15000.00

How does orchestration work?

Orchestration turns a pile of scripts into a production pipeline. A tool like Apache Airflow defines the directed acyclic graph (DAG), schedules runs, enforces task dependencies, retries failed tasks with backoff, and alerts on failure. Without orchestration you cannot reliably answer whether last night's run finished and where it stopped.

PYTHON

# Airflow DAG for the National Bank nightly pipeline
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG("bank_nightly_etl",
         start_date=datetime(2024, 1, 1),
         schedule_interval="0 2 * * *",  # 2 AM daily
         catchup=False) as dag:

    extract = PythonOperator(
        task_id="extract_transactions",
        python_callable=extract_from_kafka)

    transform = PythonOperator(
        task_id="clean_and_enrich",
        python_callable=clean_and_join_customers)

    load = PythonOperator(
        task_id="load_to_warehouse",
        python_callable=upsert_to_redshift)

    extract >> transform >> load

What is the difference between data engineering and data science?

Data engineering sits upstream of data science. Data engineers build the pipelines that collect, clean, and serve data. Data scientists consume that clean data to build models, run experiments, and produce insights. Without reliable engineering, the scientist spends most of their time cleaning data instead of modelling.

| Aspect | Data Engineering | Data Science |
|---|---|---|
| Focus | Build and operate pipelines | Analyse data and build models |
| Tools | Spark, Kafka, Airflow, SQL | Python, R, TensorFlow, Jupyter |
| Output | Clean, query-ready tables | Models, predictions, insights |
| Skill | Distributed systems, SQL, Java | Statistics, ML, domain knowledge |
| Metric | Pipeline uptime, data freshness | Model accuracy, business impact |

JAVA

package com.bank.pipeline;

// Data engineer builds the pipeline
public class PipelineBuilder {
    public void buildDailyTransactionPipeline() {
        extract();    // Pull raw txns from Kafka
        transform();  // Clean, deduplicate, enrich
        load();       // Upsert into warehouse
    }
    private void extract()   { /* Kafka consumer */ }
    private void transform() { /* Filter + join */ }
    private void load()      { /* JDBC upsert */ }
}

// Data scientist consumes the clean data
// SELECT account_id, AVG(amount) FROM clean_transactions
// GROUP BY account_id
// -> feeds into a fraud model

What does a data pipeline look like end to end?

A complete banking pipeline reads raw transaction events from Kafka, validates each event against a schema, drops or quarantines bad records, enriches each surviving record by joining to the customer reference table, and upserts the result into the warehouse keyed by transaction id.

JAVA

package com.bank.pipeline;

import java.math.BigDecimal;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

public class EndToEndPipeline {
    record RawTxn(String txnId, String accountId,
                  BigDecimal amount) {}
    record EnrichedTxn(String txnId, String accountId,
                       String holderName, BigDecimal amount) {}

    static Map<String, String> customerRef = Map.of(
        "SAV2001", "Rajesh Kumar",
        "CUR3002", "Priya Sharma"
    );

    public static void main(String[] args) {
        // 1. Extract
        List<RawTxn> raw = List.of(
            new RawTxn("T1", "SAV2001", new BigDecimal("15000")),
            new RawTxn("T2", "SAV2001", null),
            new RawTxn("T3", "CUR3002", new BigDecimal("25000"))
        );

        // 2. Transform: validate + enrich
        List<EnrichedTxn> enriched = raw.stream()
            .filter(t -> t.amount() != null)
            .filter(t -> t.amount().compareTo(BigDecimal.ZERO) > 0)
            .map(t -> new EnrichedTxn(
                t.txnId(), t.accountId(),
                customerRef.getOrDefault(t.accountId(), "UNKNOWN"),
                t.amount()))
            .collect(Collectors.toList());

        // 3. Load (upsert by txnId)
        enriched.forEach(t -> System.out.println(
            "UPSERT " + t.txnId() + " | "
            + t.holderName() + " | " + t.amount()));
    }
}

TEXT

UPSERT T1 | Rajesh Kumar | 15000
UPSERT T3 | Priya Sharma | 25000

What are the common tools in data engineering?

The ecosystem is broad, but the core categories are consistent across banks and fintech companies.

Ingestion: Apache Kafka, AWS Kinesis, Debezium (CDC), JDBC extractors.
Processing: Apache Spark, Apache Flink, Apache Beam.
Storage: S3/ADLS (lake), Snowflake/Redshift/BigQuery (warehouse), Delta Lake/Iceberg (lakehouse).
Orchestration: Apache Airflow, Dagster, Prefect.
Data quality: Great Expectations, dbt tests, custom validation.
Schema management: Confluent Schema Registry, AWS Glue Catalog.

TEXT

# Typical banking data platform stack
Sources:  Core Banking (PostgreSQL) | Card Switch (Kafka) | CIBIL API
              |                           |                    |
              v                           v                    v
Ingestion: Debezium CDC              Kafka Consumer        REST Client
              |                           |                    |
              +----------+----------------+--------------------+
                         |
                         v
Processing:         Apache Spark (clean, join, aggregate)
                         |
                         v
Storage:   S3 Data Lake (raw) --> Delta Lake (curated) --> Redshift (serving)
                         |
                         v
Orchestration:      Apache Airflow (schedule, retry, alert)

What separates a junior from a senior data engineer?

A senior data engineer obsesses over reliability and quality. The senior concerns include:

Idempotency: re-running a job must not double-count Rajesh's transfer. Upserts keyed by transaction id guarantee this.
Schema evolution: a new column appears tomorrow. The pipeline must handle it without breaking.
Late and out-of-order data: a transaction arrives after the window closes. The pipeline must accommodate it or quarantine it.
Backfills: re-processing a month of historical data must be safe and efficient.
Monitoring and alerting: per-stage row counts, latency, error rates, and lag.
Data contracts: formal agreements with upstream teams about schema, freshness, and quality.

JAVA

package com.bank.pipeline;

import java.math.BigDecimal;

public class IdempotentUpsert {
    // Upsert by txnId: re-running never creates duplicates
    static final String UPSERT_SQL = """
        INSERT INTO warehouse.transactions (txn_id, account_id, amount)
        VALUES (?, ?, ?)
        ON CONFLICT (txn_id)
        DO UPDATE SET account_id = EXCLUDED.account_id,
                      amount     = EXCLUDED.amount;
        """;

    public static void main(String[] args) {
        System.out.println("Idempotent upsert SQL:");
        System.out.println(UPSERT_SQL);
        // Running this twice with the same txn_id
        // produces exactly one row, not two
    }
}

TEXT

Idempotent upsert SQL:
INSERT INTO warehouse.transactions (txn_id, account_id, amount)
VALUES (?, ?, ?)
ON CONFLICT (txn_id)
DO UPDATE SET account_id = EXCLUDED.account_id,
              amount     = EXCLUDED.amount;

Why does data engineering matter so much in banking?

In banking, a wrong number is never just an inconvenience. It can mean a customer sees the wrong balance, a regulatory report is incorrect, or a fraud model is trained on dirty data. Data engineering ensures correctness, auditability, and recoverability.

JAVA

package com.bank.pipeline;

import java.math.BigDecimal;

public class WhyItMatters {
    public static void main(String[] args) {
        BigDecimal reported = new BigDecimal("1000000.00");
        BigDecimal actual   = new BigDecimal("999999.50");
        BigDecimal diff     = reported.subtract(actual);

        System.out.println("Reported balance: " + reported);
        System.out.println("Actual balance:   " + actual);
        System.out.println("Discrepancy:      " + diff);
        System.out.println("In banking, even 0.50 INR matters"
            + " for regulatory compliance.");
    }
}

TEXT

Reported balance: 1000000.00
Actual balance:   999999.50
Discrepancy:      0.50
In banking, even 0.50 INR matters for regulatory compliance.

What are common interview follow-up questions?

How do you handle schema changes in a running pipeline?
What is the difference between ETL and ELT?
How do you ensure exactly-once processing?
What is a data contract and why does it matter?
How do you backfill a month of data without duplicates?
What monitoring do you put on a production pipeline?

Visual Overview

DIAGRAM

  ┌─────────────────────────────────────────────────────────────┐
  │                  DATA ENGINEERING PIPELINE                   │
  └─────────────────────────────────────────────────────────────┘

  Sources                    Pipeline                    Consumers
  ─────────                  ────────                    ─────────
  ┌──────────┐    Extract    ┌──────────────┐    Serve   ┌──────────┐
  │ Core DB  │──────────────▶│              │───────────▶│ Analysts │
  │(Postgres)│               │              │            └──────────┘
  └──────────┘               │   Transform  │            ┌──────────┐
  ┌──────────┐    Ingest     │   (clean,    │───────────▶│ ML Models│
  │  Kafka   │──────────────▶│   join,      │            └──────────┘
  │ (events) │               │   aggregate) │            ┌──────────┐
  └──────────┘               │              │───────────▶│Dashboards│
  ┌──────────┐    Pull       │              │            └──────────┘
  │ APIs     │──────────────▶│              │
  │ (CIBIL)  │               └──────┬───────┘
  └──────────┘                      │
                                    │ Load
                                    ▼
                      ┌─────────────────────────┐
                      │    Storage Layer         │
                      │  ┌───────┐  ┌─────────┐ │
                      │  │ Lake  │  │Warehouse│ │
                      │  │ (S3)  │  │(Redshift│ │
                      │  │ raw   │  │ clean)  │ │
                      │  └───────┘  └─────────┘ │
                      └─────────────────────────┘

DIAGRAM

  Orchestration (Airflow DAG)
  ───────────────────────────
  extract_txns ──▶ validate ──▶ clean_and_join ──▶ load_warehouse
       │               │              │                  │
       │          on failure:    on failure:         on failure:
       │          alert + DLQ    alert + DLQ         retry x3
       ▼               ▼              ▼                  ▼
  [checkpoint]   [checkpoint]   [checkpoint]        [checkpoint]

Key Takeaway

Data engineering is the discipline of building and operating the systems that collect raw data from many sources, move it reliably, clean and transform it, and store it in a shape that analysts, data scientists, and machine learning models can query quickly and trust. A data engineer owns the full pipeline: ingestion, storage, transformation, orchestration, data quality, and the reliability and cost of all of it at scale. In practice the team sits upstream of analytics and ML, and the job is to make trustworthy data available at the right place, in the right shape, at the right time.

The work breaks into five repeating responsibilities. Ingestion is getting data out of source systems, whether that is a transactional database, a Kafka event stream, an external API, or a nightly file. Storage is deciding where data lands, typically a cheap data lake first and then a column-oriented warehouse for fast analytics. Transformation is the cleaning, standardizing, joining, deduplicating, and aggregating that turns raw records into usable ones. Orchestration is scheduling and sequencing these steps with dependencies and retries so that a downstream job only runs after its upstream succeeds.

What separates a senior data engineer is obsession with reliability and quality. Idempotency ensures that re-running a job never double-counts a transfer. Schema evolution ensures a new column tomorrow does not break the pipeline. Late and out-of-order data is handled gracefully. Monitoring covers per-stage row counts, latency, error rates, and lag. Data contracts formalize agreements with upstream teams so surprises do not silently corrupt the warehouse.

In a banking context at the National Bank of India, data engineering matters enormously because a wrong number is never just an inconvenience: it can mean a customer sees the wrong balance or a regulatory report is breached. The pipeline is designed to be correct and safely re-runnable first, and fast second.

Remember one sentence: data engineering builds the reliable plumbing that turns messy raw data into clean, trustworthy, query-ready data for everyone downstream.

---