Definition
- _SUCCESS – a zero-byte marker file written by the Hadoop FileOutputCommitter after all tasks commit.
- Completion signal – its presence means the write finished in full; its absence means incomplete output.
- Orchestration gate – downstream jobs poll for _SUCCESS before consuming the output.
- Not atomic everywhere – on eventually-consistent object stores, _SUCCESS alone does not guarantee full consistency.
- Delta Lake – uses a transaction log instead, making _SUCCESS unnecessary for consistency.
When the National Bank of India finishes printing an entire batch of statements, a supervisor stamps a "COMPLETE" slip on the output tray. Downstream clerks only pick up the tray if the slip is there; if the printing crashed halfway, there is no slip and they know not to send out a half-finished batch.
Understanding the Concepts
What is the _SUCCESS file?
When Spark writes a DataFrame to a file-based sink like Parquet, CSV, or ORC, each task writes part files. After every task commits and the job finishes, the Hadoop FileOutputCommitter writes a zero-byte _SUCCESS file in the output directory.
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("SuccessDemo").getOrCreate()
txns = spark.read.parquet("/data/transactions")
high_value = txns.filter(col("amount") > 100000)
# Write the output
high_value.write.parquet("/output/high_value_transactions")
# After successful write, the directory contains:
# /output/high_value_transactions/
# _SUCCESS <-- marker file (0 bytes)
# part-00000-abc123.parquet
# part-00001-def456.parquet
# part-00002-ghi789.parquet
How do downstream workflows use _SUCCESS?
Orchestration tools poll for _SUCCESS to know when the output is safe to consume.
# Airflow sensor waiting for _SUCCESS
# from airflow.sensors.filesystem import FileSensor
#
# wait_for_output = FileSensor(
# task_id="wait_for_high_value",
# filepath="/output/high_value_transactions/_SUCCESS",
# poke_interval=60, # Check every 60 seconds
# timeout=3600,
# )
# Simple Python check
import os
def is_output_ready(path):
success_file = os.path.join(path, "_SUCCESS")
return os.path.exists(success_file)
if is_output_ready("/output/high_value_transactions"):
print("Output is complete, proceed with downstream processing")
else:
print("Output not ready, job may be running or failed")
What happens if the write fails?
If the job fails partway, the _SUCCESS file is NOT created. This is the signal for downstream systems to not consume the output.
# If the write crashes during task execution:
# /output/high_value_transactions/
# part-00000-abc123.parquet <-- may be present
# part-00001-def456.parquet <-- may be present
# _temporary/ <-- cleanup debris
# (NO _SUCCESS file!) <-- not complete
# WARNING: never treat the presence of part files as completion
# A running or failed job can leave partial files
# Only _SUCCESS (or a transactional format) is the correct gate
How does Spark handle _SUCCESS when reading?
Spark readers automatically ignore files that start with underscore or dot, so _SUCCESS never appears as data rows.
# Reading a directory with _SUCCESS: the marker is ignored
df = spark.read.parquet("/output/high_value_transactions")
df.show() # Only shows data from part files, not _SUCCESS
# Other ignored files:
# _committed_* (commit markers)
# .crc files (checksum files)
# _temporary/ (temp directory during writes)
How do you disable _SUCCESS?
You can disable the marker file if a downstream tool is confused by it.
# Disable _SUCCESS file generation
spark.conf.set(
"mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")
high_value.write.parquet("/output/no_success_marker")
# Directory will NOT contain _SUCCESS
What are the limitations of _SUCCESS?
The _SUCCESS file tells you the job finished, but it does not make the overall write atomic on every storage system.
On HDFS:
The rename-based commit is effectively atomic per task.
_SUCCESS reliably signals completion.
On S3 (object store):
S3 has no atomic rename; the classic committer can be slow or unsafe.
_SUCCESS may appear before all part files are visible (eventual consistency).
Use optimized committers (S3A committer) or transactional formats.
With Delta Lake:
Delta uses a transaction log, not _SUCCESS, for consistency.
Readers only ever see committed data through the log.
_SUCCESS is unnecessary and typically not used.
# Delta Lake: transaction log replaces _SUCCESS
high_value.write.format("delta").save("/output/delta_high_value")
# The delta transaction log at /output/delta_high_value/_delta_log/
# provides atomic commit semantics
# No need to check for _SUCCESS
How does _SUCCESS work with partitioned writes?
For partitioned writes, _SUCCESS is written at the top-level output directory, not in each partition subdirectory.
high_value.write.partitionBy("txn_date") \
.parquet("/output/partitioned")
# Directory structure:
# /output/partitioned/
# _SUCCESS <-- top-level marker
# txn_date=2024-01-15/
# part-00000-abc123.parquet
# txn_date=2024-01-16/
# part-00001-def456.parquet
What should a senior engineer know?
- _SUCCESS is a completion stamp, not a consistency guarantee on all storage systems.
- At the National Bank of India, the nightly end-of-day write used _SUCCESS to gate the reconciliation job on HDFS.
- When the table moved to Delta Lake, _SUCCESS was no longer needed because the transaction log provided stronger guarantees.
- Never treat part files alone as proof of completion; always use _SUCCESS or a transactional format.
- For cloud object stores, use optimized committers (S3A, EMRFS) that handle eventual consistency correctly.
Visual Overview
Spark Write Process
┌────────────────────────────────────────────┐
│ Task 0: write part-00000.parquet │
│ Task 1: write part-00001.parquet │
│ Task 2: write part-00002.parquet │
└────────────────────┬───────────────────────┘
│ All tasks committed
▼
┌────────────────────────────────────────────┐
│ FileOutputCommitter writes _SUCCESS │
│ (zero bytes, completion marker) │
└────────────────────┬───────────────────────┘
│
Output Directory: ▼
┌────────────────────────────────────────────┐
│ /output/ │
│ _SUCCESS <- completion! │
│ part-00000.parquet │
│ part-00001.parquet │
│ part-00002.parquet │
└────────────────────────────────────────────┘
Failed Write (no _SUCCESS) ┌────────────────────────────────────────────┐ │ Task 0: write part-00000.parquet (OK) │ │ Task 1: FAILED │ │ Task 2: write part-00002.parquet (OK) │ └────────────────────────────────────────────┘ Output Directory: ┌────────────────────────────────────────────┐ │ /output/ │ │ (NO _SUCCESS!) <- incomplete! │ │ part-00000.parquet <- DO NOT CONSUME │ │ part-00002.parquet <- partial output │ └────────────────────────────────────────────┘
Key Takeaway
The _SUCCESS file is a zero-byte marker file that Spark writes through the Hadoop FileOutputCommitter after all tasks of a write job have committed. Its whole purpose is a completion signal: its presence means the part files are all present and final, and its absence means the job failed or is still running. Downstream orchestration tools like Airflow use sensors that poll for _SUCCESS before starting the next step.
The caveat is that _SUCCESS tells you the job finished, but it does not make the write atomic on every storage system. On HDFS the rename-based commit is effectively atomic, but on eventually-consistent object stores the classic committer can be unreliable, which is why cloud setups use optimized committers. Delta Lake goes further by using a transaction log that guarantees readers only see committed data, making _SUCCESS unnecessary.
At the National Bank of India, the nightly end-of-day write used _SUCCESS to gate the reconciliation job. When the table moved to Delta Lake, the team stopped relying on the marker because the transaction log provided stronger consistency guarantees. The important rule is to never treat part files alone as proof of completion; always use _SUCCESS or a transactional format to gate downstream consumption.
Spark readers ignore files beginning with underscore or dot, so _SUCCESS never shows up as data rows, and you can disable the marker with the marksuccessfuljobs setting if needed.
Remember one sentence: _SUCCESS is the completion stamp that tells downstream workflows the write finished in full, and you graduate to a transactional format like Delta Lake when you need true atomic consistency.
---