Definition
- Batch engine – processes a bounded dataset in scheduled runs, optimizing for throughput and cost (e.g., Spark batch).
- Micro-batch – streaming that groups events into tiny bounded batches processed at short intervals, giving high throughput with a small latency floor (e.g., Spark Structured Streaming).
- Record-at-a-time – streaming that processes each event individually as it arrives, giving the lowest latency and finer-grained state (e.g., Apache Flink).
- Trigger interval – the micro-batch frequency; the latency floor of Structured Streaming.
- Checkpoint barriers – markers Flink injects into the stream to snapshot state for exactly-once fault tolerance.
- Catalyst optimizer – Spark's query planner that micro-batch reuses from batch, a key operational advantage.
The National Bank of India can total transactions three ways. A batch engine is the night shift totalling the whole day's pile at once. Spark Structured Streaming's micro-batch is like a clerk who scoops up whatever slips landed in the last half-second and processes that little scoop, then the next scoop: continuous but in tiny stacks. Flink's record-at-a-time is a clerk who processes each slip the exact instant it touches the counter, never waiting to form a stack. Night-shift batch is cheapest for big reports; scoops are a great balance for most streaming; and per-slip is the choice when every millisecond counts, like blocking a fraudulent swipe instantly.
Understanding the Concepts
How does a batch engine work?
A batch engine like Spark batch reads an entire bounded dataset, builds an optimized plan through Catalyst, runs it across executors, writes the result, and stops until the next scheduled run. Because it sees all the data, it does large joins and full aggregations efficiently and is trivial to re-run.
# Spark batch: process the whole day at once
df = spark.read.parquet(
"s3://national-bank-lake/raw/txns/dt=2024-10-29/")
result = df.groupBy("account_id") \
.agg({"amount": "sum"}) \
.orderBy("sum(amount)", ascending=False)
result.write.parquet(
"s3://national-bank-lake/curated/daily_summary/")
+----------+-----------+
|account_id|sum(amount)|
+----------+-----------+
| CUR3002| 1250000.00|
| SAV2001| 450000.00|
+----------+-----------+
How does Spark Structured Streaming (micro-batch) work?
Spark Structured Streaming uses micro-batch: it treats the stream as a series of small bounded batches, triggering every short interval. Under the hood it reuses Spark's batch engine and Catalyst optimizer on tiny slices. This gives high throughput and the familiar DataFrame API, but carries a latency floor equal to the trigger interval.
# Spark Structured Streaming: micro-batch on Kafka
df = spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers", "kafka:9092") \
.option("subscribe", "bank-transactions") \
.load()
from pyspark.sql.functions import window, col, sum as _sum
agg = df.groupBy(
window(col("timestamp"), "1 minute"),
col("account_id")
).agg(_sum("amount").alias("total"))
agg.writeStream \
.outputMode("update") \
.format("delta") \
.option("checkpointLocation", "/checkpoints/txn_agg") \
.trigger(processingTime="2 seconds") \
.start()
# Latency floor: ~2 seconds
How does Flink (record-at-a-time) work?
Flink uses a true record-at-a-time model: each event flows through operators individually, so latency can be in the low milliseconds. This gives precise event-time handling, rich keyed state updated per event, and exactly-once via checkpoint barriers.
package com.bank.processing;
// Flink: record-at-a-time processing
// import org.apache.flink.streaming.api.*;
public class FlinkRecordAtATime {
public static void main(String[] args) {
// StreamExecutionEnvironment env =
// StreamExecutionEnvironment.getExecutionEnvironment();
//
// env.addSource(new FlinkKafkaConsumer<>(...))
// .keyBy(txn -> txn.getAccountId())
// .window(TumblingEventTimeWindows.of(Time.minutes(1)))
// .aggregate(new AmountSumAggregator())
// .addSink(new JdbcSink(...));
//
// env.execute("bank-fraud-detector");
System.out.println("Flink processes each event individually.");
System.out.println("Latency: low milliseconds.");
System.out.println("State: per-key, checkpointed, exactly-once.");
}
}
Flink processes each event individually.
Latency: low milliseconds.
State: per-key, checkpointed, exactly-once.
How do the three engines compare?
| Aspect | Spark Batch | Spark Structured Streaming | Flink |
|---|---|---|---|
| Model | Bounded, scheduled | Micro-batch (tiny slices) | Record-at-a-time |
| Latency | Minutes to hours | Hundreds of ms to seconds | Low milliseconds |
| Throughput | Very high | Very high | High |
| API | DataFrame, SQL | Same as batch | DataStream, Table API |
| Optimizer | Catalyst + Tungsten | Reuses Catalyst | Custom streaming optimizer |
| State | None (re-read) | Managed state store | RocksDB / memory backend |
| Fault tolerance | Re-run job | Checkpoints + WAL | Checkpoint barriers |
| Best for | Reports, backfills | Near-real-time dashboards | Fraud, real-time scoring |
package com.bank.processing;
public class EngineComparison {
public static void main(String[] args) {
System.out.println("Spark Batch: whole day, nightly, cheap");
System.out.println("Spark Streaming: micro-batch, ~2s latency");
System.out.println("Flink: per-event, ms latency, complex");
}
}
Spark Batch: whole day, nightly, cheap
Spark Streaming: micro-batch, ~2s latency
Flink: per-event, ms latency, complex
When do you use micro-batch vs record-at-a-time?
Use micro-batch when near-real-time of a second or two is good enough and you value one unified API and operational familiarity. Use record-at-a-time when you need the lowest latency, sophisticated per-event stateful logic, or the strongest event-time semantics.
package com.bank.processing;
public class WhenToChoose {
public static void main(String[] args) {
System.out.println("Micro-batch (Spark Streaming):");
System.out.println(" Near-real-time dashboards");
System.out.println(" Enrichment pipelines");
System.out.println(" Teams already using Spark batch");
System.out.println();
System.out.println("Record-at-a-time (Flink):");
System.out.println(" Fraud detection (block in ms)");
System.out.println(" Complex event processing (CEP)");
System.out.println(" Precise event-time windowing");
System.out.println(" Large keyed state (RocksDB)");
}
}
Micro-batch (Spark Streaming):
Near-real-time dashboards
Enrichment pipelines
Teams already using Spark batch
Record-at-a-time (Flink):
Fraud detection (block in ms)
Complex event processing (CEP)
Precise event-time windowing
Large keyed state (RocksDB)
How has the line blurred between engines?
Spark narrowed its latency with a continuous processing mode, and Flink supports batch over bounded streams, so the decision often comes down to your latency budget, how complex your state is, and which engine your team already operates well.
package com.bank.processing;
public class BlurredLines {
public static void main(String[] args) {
System.out.println("Spark moving toward streaming:");
System.out.println(" Continuous mode (experimental)");
System.out.println(" Shorter trigger intervals");
System.out.println();
System.out.println("Flink moving toward batch:");
System.out.println(" BATCH execution mode for bounded data");
System.out.println(" Unified DataStream + Table API");
System.out.println();
System.out.println("Decision factors:");
System.out.println(" 1. Latency budget");
System.out.println(" 2. State complexity");
System.out.println(" 3. Team familiarity");
}
}
Spark moving toward streaming:
Continuous mode (experimental)
Shorter trigger intervals
Flink moving toward batch:
BATCH execution mode for bounded data
Unified DataStream + Table API
Decision factors:
1. Latency budget
2. State complexity
3. Team familiarity
What does the bank use in practice?
A bank typically runs batch for reporting, micro-batch for most near-real-time dashboards and enrichment, and reserves Flink for the latency-critical, heavily stateful fraud and risk jobs. Matching the engine to the workload keeps the platform both cost-effective and fast where it matters.
package com.bank.processing;
public class BankPractice {
public static void main(String[] args) {
System.out.println("National Bank engine allocation:");
System.out.println(" Spark Batch: nightly reports, backfills");
System.out.println(" Spark Streaming: near-RT dashboards");
System.out.println(" Flink: fraud detection, risk scoring");
System.out.println();
System.out.println("Result: cost-effective + fast where needed.");
}
}
National Bank engine allocation:
Spark Batch: nightly reports, backfills
Spark Streaming: near-RT dashboards
Flink: fraud detection, risk scoring
Result: cost-effective + fast where needed.
What are common interview follow-up questions?
- What is the latency floor of Spark Structured Streaming and why?
- How does Flink achieve exactly-once with checkpoint barriers?
- When would you use Spark's continuous mode over micro-batch?
- How do you migrate a batch Spark job to Structured Streaming?
- How do you choose between Flink and Kafka Streams?
Visual Overview
Three Processing Models
───────────────────────
Spark Batch (bounded, scheduled)
[Day's data] ──▶ [Optimize] ──▶ [Run all] ──▶ [Result] ──▶ STOP
Spark Structured Streaming (micro-batch)
[Events] ──▶ [Scoop 0.5s] ──▶ [Process] ──▶ [Scoop] ──▶ [Process] ──▶ ...
└─ trigger ─┘ └─ result ─┘
Flink (record-at-a-time)
[Event] ──▶ [Process] ──▶ [Event] ──▶ [Process] ──▶ [Event] ──▶ ...
(each event processed individually, ms latency)
Latency Spectrum ──────────────── Batch Micro-Batch Record-at-a-Time (hours) (seconds) (milliseconds) ◄────────────────────────────────────────────────► Spark Batch Spark Streaming Flink Cheapest Good balance Lowest latency
Key Takeaway
A batch engine like Spark batch processes a bounded dataset in scheduled runs, reading the whole chunk, building an optimized plan, running it, and stopping. This makes it efficient for large joins and aggregations and ideal for end-of-day reports and backfills. Spark Structured Streaming uses micro-batch, treating the stream as tiny bounded batches triggered every short interval, reusing Spark's Catalyst optimizer and DataFrame API for high throughput with a latency floor of roughly the trigger interval. Flink uses true record-at-a-time processing, flowing each event through operators individually for low-millisecond latency, with precise event-time handling and exactly-once via checkpoint barriers.
Choosing between them comes down to latency, throughput, state complexity, and operational simplicity. Use batch when freshness is not critical and cost wins. Use micro-batch when near-real-time of a second or two is good enough and you value one unified API. Use record-at-a-time when you need the lowest latency, sophisticated per-event stateful logic, or the strongest event-time semantics.
The line has blurred, since Spark added continuous mode and Flink supports batch over bounded streams, so the real decision is your latency budget, how complex your state is, and which engine your team already operates well.
In a banking context at the National Bank of India, nightly reports ran on Spark batch, near-real-time dashboards ran on Spark Structured Streaming, and the latency-critical fraud jobs ran on Flink where processing each swipe instantly let the bank block a fraudulent transaction immediately. Matching the engine to the workload is what kept the platform cost-effective and fast where it mattered.
Remember one sentence: batch for throughput, micro-batch for near-real-time balance, record-at-a-time for lowest latency, and match the engine to the workload.
---