Definition
- pathGlobFilter – a DataFrameReader option that restricts which files are read by matching a glob pattern against the file name, applied during file listing.
- Partition discovery – Spark's automatic mechanism that reads Hive-style directory structure (e.g.,
year=2026/month=10) and turns folder names into columns used for partition pruning. - Independence – the two features operate on different things and compose cleanly; the glob narrows files, discovery derives columns.
- Cost saving – on object stores, filtering during listing avoids opening and reading irrelevant files, saving both time and cost.
The National Bank of India keeps a storage room full of mixed documents: trade files ending in .csv, scanned images ending in .jpg, and temp files ending in .tmp. pathGlobFilter is the instruction to the clerk: only pick up files whose name ends in .csv. Partition discovery is a completely different idea: it is the clerk reading the labeled shelf structure – Year 2026 shelf, October drawer – and tagging each file with which shelf and drawer it came from, so you can later say "only the October drawer" and skip the rest.
Understanding the Concepts
What problem does pathGlobFilter solve?
When you point Spark at a directory, it tries to read every file under that path. Often the directory contains files you do not want: markers, manifests, log files, or files from another format.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("GlobDemo").getOrCreate()
# The landing directory contains mixed files:
# /lake/landing/trades_2026-10-30.csv
# /lake/landing/trades_2026-10-30_SUCCESS
# /lake/landing/trades_2026-10-30_manifest.json
# /lake/landing/upload_temp_abc123.tmp
# Without pathGlobFilter: Spark tries to parse everything as CSV → fails
txn = spark.read.csv("/lake/landing/", header=True) # error on non-CSV files
How do you use pathGlobFilter?
Set it as a reader option with a glob pattern.
# Only read files ending in .csv
txn = spark.read \
.option("pathGlobFilter", "*.csv") \
.csv("/lake/landing/", header=True)
# Only read trade files (specific prefix)
txn = spark.read \
.option("pathGlobFilter", "trades_*.csv") \
.csv("/lake/landing/", header=True)
# The pattern matches the file name only, not the full path
# Supports * (any characters) and ? (single character)
Spark lists the directory and only opens files whose names match the pattern.
How is pathGlobFilter different from partition discovery?
They operate on completely different things.
# Partition discovery reads folder structure into columns:
# /lake/trades/trade_date=2026-10-28/trades_0000.parquet
# /lake/trades/trade_date=2026-10-29/trades_0000.parquet
# /lake/trades/trade_date=2026-10-30/trades_0000.parquet
# ^^^ marker_SUCCESS (not a data file)
# pathGlobFilter filters files WITHIN each partition folder
txn = spark.read \
.option("pathGlobFilter", "*.parquet") \
.parquet("/lake/trades/")
# Partition discovery still derives trade_date from folder names
# pathGlobFilter skips the SUCCESS marker files
txn.printSchema()
root |-- trade_id: long |-- account_id: long |-- amount: decimal(18,2) |-- trade_date: date ← derived from partition folder name
Both features are active simultaneously and compose cleanly.
How do you combine pathGlobFilter with partition pruning?
Use pathGlobFilter to exclude non-data files and a filter on the partition column to prune partitions.
# pathGlobFilter: skip non-parquet files
# Partition pruning: only read October data
result = spark.read \
.option("pathGlobFilter", "*.parquet") \
.parquet("/lake/trades/") \
.filter("trade_date >= '2026-10-01' AND trade_date < '2026-11-01'")
result.explain(True)
== Physical Plan ==
*(1) FileScan parquet
PartitionFilters: [trade_date >= 2026-10-01, trade_date < 2026-11-01]
PathGlobFilter: *.parquet
Partition pruning skips entire date folders. pathGlobFilter skips non-data files within the remaining folders.
What are related file-filtering options?
Spark provides several options that each filter differently.
# recursiveFileLookup: recurse into subdirectories, disable partition discovery
spark.read \
.option("recursiveFileLookup", "true") \
.option("pathGlobFilter", "*.csv") \
.csv("/lake/flat_landing/")
# modifiedBefore / modifiedAfter: filter by file timestamp
spark.read \
.option("modifiedAfter", "2026-10-30T00:00:00") \
.option("pathGlobFilter", "*.parquet") \
.parquet("/lake/trades/")
| Option | Filters by | Affects partition discovery |
|---|---|---|
| pathGlobFilter | File name pattern | No |
| recursiveFileLookup | Recurse depth | Yes (disables discovery) |
| modifiedBefore/After | File timestamp | No |
| basePath | Discovery root | Yes (sets the root) |
What happens on object stores?
On S3 and similar stores, listing is expensive. pathGlobFilter reduces the number of files opened, which saves time and cost.
# Without glob: Spark lists and opens 10,000 files including markers
# With glob: Spark lists 10,000 but opens only the 8,000 data files
# Combined with a well-chosen base path, you avoid listing irrelevant directories too
What should a senior engineer know?
- pathGlobFilter is a file-name filter, not a data-value filter. It cannot prune by date ranges.
- For pruning by data values like dates, rely on partition discovery + filter predicates.
- Combine pathGlobFilter with partition pruning for maximum efficiency.
- On object stores, pair the glob with a narrow base path to minimize listing.
- In streaming with Auto Loader, use
cloudFiles.pathGlobFilterfor the same effect.
# Auto Loader with path glob filter
stream = spark.readStream \
.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.option("pathGlobFilter", "trades_*.csv") \
.load("/lake/landing/")
What are common interview follow-up questions?
- Does pathGlobFilter work with Parquet? Yes, with any file format reader.
- Can you use regex instead of glob? No, pathGlobFilter uses glob syntax only (*, ?).
- Does it affect how many files are listed? No, Spark still lists the directory. It affects which listed files are opened and read.
Visual Overview
pathGlobFilter vs Partition Discovery
──────────────────────────────────────
Directory: /lake/trades/
├── trade_date=2026-10-28/
│ ├── trades_0000.parquet ← read (matches glob)
│ ├── trades_0001.parquet ← read
│ └── _SUCCESS ← SKIPPED (glob: *.parquet)
├── trade_date=2026-10-29/
│ ├── trades_0000.parquet ← read
│ └── manifest.json ← SKIPPED
└── trade_date=2026-10-30/ ← PRUNED (partition filter)
├── trades_0000.parquet
└── _SUCCESS
pathGlobFilter: *.parquet → skips non-parquet files
Partition filter: < 10-30 → skips entire 10-30 folder
Both active simultaneously, composing cleanly
Key Takeaway
pathGlobFilter is a Spark read option that restricts which files are read from a directory by matching a glob pattern against the file name. You set it with a pattern like *.csv or trades_*.parquet, and Spark only opens files whose names match, skipping markers, manifests, temp files, and anything else in the directory. It is applied during file listing, so on object stores where listing is expensive, it saves both time and cost.
The key distinction is that pathGlobFilter is not partition discovery. Partition discovery reads Hive-style folder names like trade_date=2026-10-30 and derives partition columns for pruning. pathGlobFilter never touches directory structure or partition columns. The two features operate independently and compose cleanly: the glob filters which files within each partition are read, while discovery still derives the partition column from the folder names.
At the National Bank of India, each date folder in the trades landing zone contained not just trade CSVs but also SUCCESS markers, manifests, and occasional temp files from interrupted uploads. The reader kept failing because Spark tried to parse non-data files as CSV. Setting pathGlobFilter to the trade file pattern made Spark read only the data files, and partition discovery still derived the trade date from the folder names. The rule is to use pathGlobFilter for excluding non-data files and partition predicates for pruning by data values.
Remember one sentence: pathGlobFilter filters files by name pattern during listing, partition discovery derives columns from folder names for pruning, and the two are independent and compose cleanly.
---