The three output modes in Structured Streaming

• Append mode – Emits only rows that are new and will never change again, suited for stateless transforms and watermark-finalized aggregations; the only mode file sinks support.
• Complete mode – Rewrites the entire result table to the sink every trigger, valid only for aggregation queries where the sink needs a full snapshot each time.
• Update mode – Emits only the rows whose values changed since the last trigger, the efficient middle ground for running aggregations to an upsert-capable sink.
• Query constraint – The choice is constrained by both the query type and the sink; Spark rejects invalid pairings at query start.
• State interaction – In Append with aggregation, the watermark controls when rows become final and when state is dropped; Complete never drops state because the full table must stay re-emittable.

What are the three output modes?

Structured Streaming's output mode controls which rows of the result table are written to the sink on each trigger. You set it with .outputMode("append"), .outputMode("complete"), or .outputMode("update") on the writeStream call.

// Setting the output mode on a streaming query
counts.writeStream()
    .outputMode("update")  // append | complete | update
    .format("console")
    .start();

How does Append mode work?

Append mode writes only rows added to the result table that are guaranteed never to change again. For a stateless query like a filter or a map, every output row is final, so Append is the natural mode. For aggregations, Append can still be used but only with a watermark, because Spark can only emit an aggregate once the watermark passes the window and guarantees no later update.

import static org.apache.spark.sql.functions.*;

// Stateless filter -- Append is natural
Dataset<Row> highValue = transactions
    .filter(col("amount").gt(new java.math.BigDecimal("100000.00")))
    .select("accountId", "amount", "eventTime");

highValue.writeStream()
    .outputMode("append")
    .format("parquet")
    .option("path", "s3a://nbi-data/high-value-txns")
    .option("checkpointLocation", "s3a://nbi-checkpoints/high-value")
    .start();

Batch 0: writes 15 new high-value rows to Parquet
Batch 1: writes 8 new high-value rows to Parquet
(Old rows are never updated or re-emitted)

How does Complete mode work?

Complete mode rewrites the entire result table to the sink on every trigger. It is only valid for aggregation queries because Spark must hold the full result in state to re-emit it. You use it when the sink needs a complete snapshot each time.

// Complete mode -- full snapshot every trigger
Dataset<Row> branchTotals = transactions
    .groupBy("branchCode")
    .agg(sum("amount").as("totalAmount"));

branchTotals.writeStream()
    .outputMode("complete")
    .format("console")
    .start();

Batch 0:
+----------+-----------+
|branchCode|totalAmount|
+----------+-----------+
|    MUMBAI|  500000.00|
|     DELHI|  320000.00|
+----------+-----------+

Batch 1 (entire table rewritten):
+----------+-----------+
|branchCode|totalAmount|
+----------+-----------+
|    MUMBAI|  750000.00|
|     DELHI|  410000.00|
|   CHENNAI|  120000.00|
+----------+-----------+

How does Update mode work?

Update mode writes only the rows whose value changed since the previous trigger. For a running groupBy("accountId").count(), if only three accounts received transactions in this micro-batch, only those three rows are emitted. This is the efficient choice for incremental aggregations going to an upsert-capable sink.

// Update mode -- only changed rows emitted
Dataset<Row> accountCounts = transactions
    .groupBy("accountId")
    .count();

accountCounts.writeStream()
    .outputMode("update")
    .format("jdbc")
    .option("url", "jdbc:postgresql://nbi-db:5432/analytics")
    .option("dbtable", "account_txn_counts")
    .start();

Batch 0: emits 1000 account rows (all new)
Batch 1: emits 45 rows (only accounts that received new transactions)
Batch 2: emits 12 rows (only accounts that changed)

How do you choose between the three modes?

The choice is constrained by both the query and the sink.

| Scenario | Mode | Reason |
|---|---|---|
| Stateless transform to files (S3/HDFS) | Append | File sinks support only Append |
| Aggregation to key-value store / dashboard | Complete | Sink needs full snapshot |
| Aggregation to database with upsert | Update | Minimal data transfer |
| Windowed aggregation to files | Append + watermark | Rows finalized by watermark |
| Small reference table refresh | Complete | Result is small, full rewrite is acceptable |

// File sink -- MUST use Append
transactions.writeStream().outputMode("append").format("parquet").start();

// Aggregation without watermark -- CANNOT use Append
// This fails with AnalysisException:
transactions.groupBy("accountId").count()
    .writeStream().outputMode("append").start(); // ERROR

What are the rules and constraints?

Spark fails fast at query start if the output mode is invalid for the query type. A file sink forces Append. An aggregation without a watermark cannot use Append because Spark cannot know an aggregate is final. Complete needs an aggregation and a sink that accepts a full rewrite. Update needs a sink that can upsert by key. Getting the pairing wrong produces an AnalysisException.

try {
    // Invalid: Complete mode on a stateless query
    transactions.filter(col("amount").gt(0))
        .writeStream()
        .outputMode("complete") // AnalysisException
        .start();
} catch (Exception e) {
    System.out.println(e.getMessage());
    // Complete output mode not supported for non-aggregation queries
}

How does output mode interact with state and watermarks?

In Append mode with aggregation, the watermark both decides when a row becomes final and when its state can be dropped, so without a watermark the state grows unbounded and Append is rejected. In Complete mode state is never dropped because the full table must always be re-emittable, so Complete is only safe when the number of groups is bounded and small. In Update mode, the watermark still governs when state is dropped and when late events are ignored.

// Append with watermark -- rows emitted when window finalizes
Dataset<Row> windowed = transactions
    .withWatermark("eventTime", "10 minutes")
    .groupBy(window(col("eventTime"), "5 minutes"), col("accountId"))
    .agg(sum("amount").as("total"));

windowed.writeStream()
    .outputMode("append")  // valid because watermark finalizes windows
    .format("parquet")
    .option("path", "s3a://nbi-data/windowed-totals")
    .option("checkpointLocation", "s3a://nbi-checkpoints/windowed")
    .start();

What are common mistakes?

The most common mistake is using Complete mode for a large aggregation where the result table has millions of groups. Every trigger rewrites all groups, which does not scale. The correct approach is Update mode with an upsert sink.

// WRONG -- Complete mode with millions of accounts
// Every trigger writes millions of rows to the sink
transactions.groupBy("accountId").count()
    .writeStream()
    .outputMode("complete") // Writes ALL accounts every trigger
    .format("console")
    .start();

// CORRECT -- Update mode, only changed accounts emitted
transactions.groupBy("accountId").count()
    .writeStream()
    .outputMode("update") // Writes only changed accounts
    .format("console")
    .start();

What should a senior engineer know?

Choosing the output mode is also a choice about memory and correctness, not just output volume. Complete mode holds all state forever and rewrites everything. Append mode with watermarks gives eventual finality and state cleanup. Update mode gives the smallest output footprint but requires the sink to handle upserts. In banking pipelines, Update mode with a PostgreSQL upsert on account id is the standard pattern for live per-account dashboards.