Back to Advanced Data Engineering
Q5Question 5 of 24

What is Auto Loader (`cloudFiles`)

Key Concepts
  • Auto Loader is a Databricks streaming source using the cloudFiles format that incrementally ingests only new files, exactly once.
  • It tracks processed files in a streaming checkpoint so on restart it resumes exactly where it left off.
  • Directory listing mode lists the path and diffs against the checkpoint; works everywhere but gets costly at scale.
  • File notification mode subscribes to cloud-native events (SNS/SQS, Event Grid) so it scales without listing overhead.
  • Schema management: Auto Loader can infer, persist, and evolve the schema and rescue unexpected data into a rescued column.
  • It pairs naturally with the medallion architecture, landing raw files into a Bronze Delta table.

Definition

Auto Loader incrementally ingests only the new files arriving in cloud storage, exactly once, using checkpointing plus directory listing or file notifications, so it scales to millions of files without reprocessing.
  • What it is – a Databricks Structured Streaming source invoked with the cloudFiles format that incrementally ingests new files as they land in cloud object storage.
  • Exactly-once – tracks processed files in a streaming checkpoint, so each file is processed once and none is skipped, even across restarts.
  • Two discovery modes – directory listing (no setup, costly at scale) and file notification (cloud-native events, scalable but needs permissions).
  • Schema management – can infer, persist, and evolve the schema automatically and rescue unexpected data into a rescued column.

The National Bank of India has a mail slot where branches continuously drop new trade statements as files. A naive approach re-reads the entire mail room every time, which gets slower as the pile grows and risks processing the same statement twice. Auto Loader is like a diligent mail clerk who keeps a precise logbook of exactly which envelopes have been opened, so each time he only picks up the new ones, opens each exactly once, and never loses his place even if he goes home and comes back. He can either walk the room to see what is new, or get a buzzer every time a new envelope drops.

Understanding the Concepts

What problem does Auto Loader solve?

A common data engineering task is ingesting files that keep arriving in a cloud storage folder. The naive batch approach re-reads the whole folder each run and diffs to find new files, which is slow, error-prone, and gets worse as file counts grow.

PYTHON
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("AutoLoaderDemo").getOrCreate()

# Naive batch approach: slow, error-prone, does not scale
all_files = spark.read.csv("/lake/landing/trades/", header=True)
# How do you know which files are new? How do you avoid reprocessing?

Auto Loader is purpose-built for this.

How do you use Auto Loader?

Read with the cloudFiles format inside a streaming query.

PYTHON
# Auto Loader incrementally picks up only new files
stream = spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "csv") \
    .option("cloudFiles.schemaLocation", "/checkpoints/trades_schema") \
    .option("header", "true") \
    .load("/lake/landing/trades/")

# Write to Bronze Delta table
stream.writeStream \
    .format("delta") \
    .option("checkpointLocation", "/checkpoints/trades_bronze") \
    .trigger(availableNow=True) \
    .start("/lake/bronze/trades")

The checkpoint records which files have been processed. On restart, Auto Loader resumes exactly where it left off.

What are the two file-discovery modes?

Auto Loader discovers new files using either directory listing or file notification.

PYTHON
# Directory listing mode (default):
# Lists the storage path and diffs against the checkpoint
# No extra setup, works everywhere, but listing is expensive at scale
stream = spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "csv") \
    .option("cloudFiles.useNotifications", "false") \
    .load("/lake/landing/trades/")

# File notification mode:
# Subscribes to cloud-native events (AWS SNS+SQS, Azure Event Grid)
# Scales to very high file volumes without listing cost
stream = spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "csv") \
    .option("cloudFiles.useNotifications", "true") \
    .load("/lake/landing/trades/")

Notification mode can be auto-provisioned by Auto Loader when given the right cloud permissions.

How does schema management work?

Auto Loader can infer the schema from incoming files, persist it, and evolve it when new columns appear.

PYTHON
# Schema inference and persistence
stream = spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "csv") \
    .option("cloudFiles.schemaLocation", "/checkpoints/trades_schema") \
    .option("cloudFiles.schemaEvolutionMode", "addNewColumns") \
    .option("header", "true") \
    .load("/lake/landing/trades/")

# When a new column appears in the source files:
# - Auto Loader detects it
# - Adds it to the persisted schema
# - Includes it in subsequent reads

# Unexpected data is rescued into a special column
stream = spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "csv") \
    .option("cloudFiles.schemaLocation", "/checkpoints/trades_schema") \
    .option("rescuedDataColumn", "_rescued_data") \
    .load("/lake/landing/trades/")

How does Auto Loader pair with the medallion architecture?

Auto Loader lands raw files into Bronze; downstream jobs clean to Silver and aggregate to Gold.

PYTHON
# Bronze: Auto Loader ingests raw files
bronze_stream = spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "csv") \
    .option("cloudFiles.schemaLocation", "/checkpoints/trades_schema") \
    .option("header", "true") \
    .load("/lake/landing/trades/")

from pyspark.sql import functions as F

bronze_stream \
    .withColumn("_ingest_ts", F.current_timestamp()) \
    .withColumn("_source_file", F.input_file_name()) \
    .writeStream \
    .format("delta") \
    .option("checkpointLocation", "/checkpoints/bronze_trades") \
    .start("/lake/bronze/trades")

# Silver and Gold process the Bronze Delta table downstream

What are the operational requirements?

The checkpoint and schema location must be preserved for exactly-once guarantees and schema continuity.

PYTHON
# Critical: do not delete the checkpoint or schema location
# If the checkpoint is lost, Auto Loader cannot determine which files
# were already processed and may reprocess or skip files

# For notification mode, grant the required cloud permissions:
# AWS: permissions to create SNS topics, SQS queues, S3 event notifications
# Azure: permissions to create Event Grid subscriptions and Storage queues

How does Auto Loader compare to a manual file-tracking approach?

| Feature | Auto Loader | Manual file tracking |
|---|---|---|
| Exactly-once | Built-in via checkpoint | Must implement yourself |
| Scalability | Notification mode scales to millions | Listing gets slow |
| Schema evolution | Automatic | Manual schema management |
| Restart safety | Resume from checkpoint | Must track state yourself |
| Operational cost | Minimal | Significant custom code |

What should a senior engineer know?

  • Auto Loader replaces fragile hand-rolled file tracking for any continuously arriving feed.
  • Start with directory listing mode for simplicity; switch to notifications when file volumes grow.
  • Preserve the checkpoint and schema location; they are the source of truth for processing state.
  • Pair with Delta Live Tables for declarative, production-grade pipeline management.
  • Use trigger(availableNow=True) for batch-like behavior with streaming exactly-once guarantees.
PYTHON
# Production pattern: Auto Loader with exactly-once batch trigger
stream.writeStream \
    .format("delta") \
    .option("checkpointLocation", "/checkpoints/bronze_trades") \
    .trigger(availableNow=True) \
    .start("/lake/bronze/trades")

What are common interview follow-up questions?

  • Does Auto Loader work outside Databricks? The cloudFiles source is Databricks-specific. On open-source Spark, use a custom directory-listing approach or a file streaming source.
  • Can Auto Loader handle nested directories? Yes, it recursively discovers files under the given path.
  • What happens if a file is modified after processing? Auto Loader tracks file paths, not content. A modified file is not reprocessed unless it has a new path.

Visual Overview

DIAGRAM
  Auto Loader Pipeline
  ─────────────────────

  Cloud Storage (S3 / ADLS / GCS)
  ┌──────────────────────────────────┐
  │  /landing/trades/                │
  │  ├── trades_2026-10-28.csv  [ok] │
  │  ├── trades_2026-10-29.csv  [ok] │
  │  └── trades_2026-10-30.csv  [NEW]│
  └──────────────────┬───────────────┘
                     │
    Auto Loader (cloudFiles)
    ┌────────────────┴────────────────┐
    │ 1. Discover new files           │
    │    (listing or notifications)   │
    │ 2. Check against checkpoint     │
    │ 3. Read only new files          │
    │ 4. Apply schema (infer/evolve)  │
    │ 5. Update checkpoint            │
    └────────────────┬────────────────┘
                     │
  ┌──────────────────▼───────────────┐
  │  Bronze Delta Table              │
  │  (exactly-once, append-only)     │
  └──────────────────────────────────┘

Key Takeaway

Auto Loader is a Databricks Structured Streaming source, used with the cloudFiles format, that incrementally ingests new files as they land in cloud storage and processes each file exactly once. The problem it solves is that re-reading a whole folder each run and diffing to find new files is slow, error-prone, and gets worse as file counts grow. Auto Loader keeps the set of processed files in a streaming checkpoint, so each trigger picks up only the new arrivals, and on restart it resumes exactly where it left off.

It discovers files in two modes: directory listing, which needs no setup but costs more at scale, and file notification, which subscribes to cloud-native events for push-based discovery that scales without listing overhead. On top of discovery, Auto Loader handles schema management: inferring, persisting, and evolving the schema as new columns appear, and rescuing unexpected data instead of failing.

At the National Bank of India, branches continuously dropped trade files into a landing folder, and the original hourly pipeline re-read the whole folder and diffed it, which got slow and occasionally double-counted trades. Switching to Auto Loader meant only new files were processed, each exactly once across restarts. When we moved to notification mode, it stopped paying the listing cost and scaled comfortably. When upstream added a new fee column, schema evolution absorbed it instead of crashing the job.

Remember one sentence: Auto Loader incrementally ingests only new files with exactly-once checkpointing and automatic schema evolution, replacing fragile hand-rolled file tracking for any continuously arriving feed.

---

Share
Use ← → arrow keys to navigate