Definition
- Batch processing – collecting data over a window of time into a bounded dataset and processing the whole chunk in one scheduled run, optimizing for throughput and cost.
- Stream processing – handling an unbounded sequence of events continuously, processing each event or micro-batch as it arrives, optimizing for low latency.
- Bounded dataset – data with a known start and end, like all transactions for 2024-10-29.
- Unbounded dataset – events that keep arriving forever with no natural end.
- Micro-batch – streaming that groups events into tiny batches at short intervals (Spark Structured Streaming).
- Record-at-a-time – streaming that processes each event individually for the lowest latency (Apache Flink).
Think of the National Bank of India in two modes. Batch is like the end-of-day reconciliation: the staff wait until midnight, take the whole day's pile of transactions, and process them together in one big run. Cheaper and simpler, but you only learn today's totals tomorrow morning. Stream is like the fraud desk watching each card swipe live: the moment Rajesh's card is swiped in two cities within one minute, an alert fires immediately. Batch answers "what happened yesterday"; streaming answers "what is happening right now".
Understanding the Concepts
What is batch processing?
Batch processing works on a bounded dataset: a dataset with a known start and end, like all transactions for 2024-10-29. A scheduler kicks off a job, the job reads the entire chunk, performs transformations and aggregations, writes the result, and stops until the next scheduled run. Because the job sees all the data at once, it can do heavy operations like large joins and full re-sorts efficiently.
package com.bank.processing;
import java.math.BigDecimal;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
public class BatchExample {
record Transaction(String accountId, BigDecimal amount) {}
public static void main(String[] args) {
// Batch: process the whole day's transactions at once
List<Transaction> dailyBatch = List.of(
new Transaction("SAV2001", new BigDecimal("15000")),
new Transaction("SAV2001", new BigDecimal("5000")),
new Transaction("CUR3002", new BigDecimal("25000")),
new Transaction("CUR3002", new BigDecimal("10000")));
Map<String, BigDecimal> totals = dailyBatch.stream()
.collect(Collectors.groupingBy(
Transaction::accountId,
Collectors.reducing(BigDecimal.ZERO,
Transaction::amount, BigDecimal::add)));
totals.forEach((acct, sum) ->
System.out.println(acct + " -> " + sum));
}
}
SAV2001 -> 20000
CUR3002 -> 35000
What is stream processing?
Stream processing works on an unbounded dataset: events keep arriving forever with no natural end. A long-running job consumes events continuously and emits results continuously. Because you never see all the data, computations over time must use windows.
package com.bank.processing;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class StreamExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("group.id", "fraud-detector");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
try (KafkaConsumer<String, String> consumer =
new KafkaConsumer<>(props)) {
consumer.subscribe(
Collections.singletonList("card-swipes"));
// Continuous: never stops
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(200));
for (ConsumerRecord<String, String> r : records) {
System.out.println("Processing swipe: "
+ r.value());
// Evaluate fraud rules immediately
}
}
}
}
}
How are batch and stream processing different?
| Aspect | Batch | Stream |
|---|---|---|
| Dataset | Bounded (known start/end) | Unbounded (infinite) |
| Schedule | Periodic (hourly, daily) | Continuous |
| Latency | Minutes to hours | Milliseconds to seconds |
| Throughput | Very high (whole chunk) | High but per-event overhead |
| Complexity | Simpler (complete data) | Harder (late data, ordering) |
| Re-run | Trivial (re-read chunk) | Needs checkpoints, replay |
| Use case | Reports, backfills | Alerts, dashboards, fraud |
package com.bank.processing;
public class ComparisonSummary {
public static void main(String[] args) {
System.out.println("Batch: process 2024-10-29 at midnight");
System.out.println(" All data present. Simple. Complete.");
System.out.println();
System.out.println("Stream: process each swipe as it arrives");
System.out.println(" Immediate. Complex. Partial view.");
}
}
Batch: process 2024-10-29 at midnight
All data present. Simple. Complete.
Stream: process each swipe as it arrives
Immediate. Complex. Partial view.
What are event-time, processing-time, and watermarks?
Streaming must deal with event-time versus processing-time and late data, which batch mostly sidesteps. Event-time is when the event actually occurred; processing-time is when the system sees it. Watermarks are progress markers that tell the engine "all events up to this time have likely arrived", allowing windows to close.
package com.bank.processing;
import java.time.Instant;
public class EventTimeVsProcessingTime {
public static void main(String[] args) {
Instant eventTime = Instant.parse(
"2024-10-29T09:15:00Z"); // when swipe happened
Instant processingTime = Instant.parse(
"2024-10-29T09:15:03Z"); // when system received it
System.out.println("Event time: " + eventTime);
System.out.println("Processing time: " + processingTime);
System.out.println("Delay: 3 seconds");
System.out.println();
System.out.println("Watermark at 09:15:00 means:");
System.out.println(" All events before 09:15:00"
+ " have likely arrived.");
System.out.println(" Windows ending at 09:15:00 can close.");
}
}
Event time: 2024-10-29T09:15:00Z
Processing time: 2024-10-29T09:15:03Z
Delay: 3 seconds
Watermark at 09:15:00 means:
All events before 09:15:00 have likely arrived.
Windows ending at 09:15:00 can close.
What is micro-batch vs record-at-a-time?
Spark Structured Streaming uses micro-batch: it treats the stream as a series of tiny bounded batches triggered at short intervals. This reuses Spark's optimizer and gives high throughput with a small latency floor. Flink uses record-at-a-time: each event flows through operators individually for the lowest latency.
# Spark Structured Streaming: micro-batch
df = spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers", "kafka:9092") \
.option("subscribe", "card-swipes") \
.load()
result = df.groupBy("account_id").count()
result.writeStream \
.outputMode("complete") \
.format("console") \
.trigger(processingTime="2 seconds") \
.start()
# Latency floor: ~2 seconds (trigger interval)
# Flink: record-at-a-time
# Each card swipe processed individually
# Latency: low milliseconds
# But more complex to operate
What are Lambda and Kappa architectures?
Lambda architecture combines a batch layer for complete, correct results with a speed (streaming) layer for low-latency approximate results. Kappa architecture uses stream-only: treat everything as a stream and simplify by eliminating the separate batch layer.
package com.bank.processing;
public class LambdaVsKappa {
public static void main(String[] args) {
System.out.println("Lambda architecture:");
System.out.println(" Batch layer: nightly, complete, correct");
System.out.println(" Speed layer: streaming, fast, approximate");
System.out.println(" Serving layer: merge both for queries");
System.out.println(" Downside: maintain two codebases");
System.out.println();
System.out.println("Kappa architecture:");
System.out.println(" Stream-only: everything is a stream");
System.out.println(" Re-process by replaying the log");
System.out.println(" Simpler: one codebase");
System.out.println(" Downside: harder for some batch jobs");
}
}
Lambda architecture:
Batch layer: nightly, complete, correct
Speed layer: streaming, fast, approximate
Serving layer: merge both for queries
Downside: maintain two codebases
Kappa architecture:
Stream-only: everything is a stream
Re-process by replaying the log
Simpler: one codebase
Downside: harder for some batch jobs
When should you use batch vs streaming?
Use batch when freshness is not critical and cost and simplicity win, like nightly reports and backfills. Use streaming when the business needs immediate results, like fraud detection and live dashboards. Many banks use both on the same data.
package com.bank.processing;
public class WhenToUse {
public static void main(String[] args) {
System.out.println("Use BATCH for:");
System.out.println(" - Nightly reconciliation");
System.out.println(" - Historical backfills");
System.out.println(" - Regulatory reports");
System.out.println(" - Cost-sensitive aggregations");
System.out.println();
System.out.println("Use STREAMING for:");
System.out.println(" - Fraud detection (immediate)");
System.out.println(" - Live dashboards");
System.out.println(" - Real-time notifications");
System.out.println(" - Event-driven microservices");
}
}
Use BATCH for:
- Nightly reconciliation
- Historical backfills
- Regulatory reports
- Cost-sensitive aggregations
Use STREAMING for:
- Fraud detection (immediate)
- Live dashboards
- Real-time notifications
- Event-driven microservices
What are common interview follow-up questions?
- How do you handle late-arriving data in a streaming pipeline?
- What is the difference between tumbling and sliding windows?
- How does exactly-once processing work in streaming?
- When would you choose Kappa over Lambda architecture?
- How do you test a streaming pipeline?
Visual Overview
Batch vs Stream Processing ────────────────────────── Batch Processing ┌──────────┐ midnight ┌──────────────┐ result ┌──────────┐ │ Day's │─────────────▶│ Process ALL │──────────▶│ Daily │ │ events │ (scheduled) │ at once │ │ report │ │ (bounded)│ │ (joins, aggs)│ │ │ └──────────┘ └──────────────┘ └──────────┘ Stream Processing ┌──────────┐ continuous ┌──────────────┐ immediate ┌──────────┐ │ Each │─────────────▶│ Process each │──────────▶│ Alert / │ │ event │ (as it │ event now │ │ dashboard│ │(unbounded│ arrives) │ (windows) │ │ │ └──────────┘ └──────────────┘ └──────────┘
Lambda Architecture
┌──────────────────────────────┐
│ Raw Event Log │
└──────┬──────────────┬────────┘
│ │
Batch Layer Speed Layer
(nightly) (streaming)
│ │
▼ ▼
┌──────────────────────────────┐
│ Serving Layer │
│ (merge batch + stream) │
└──────────────────────────────┘
Key Takeaway
Batch processing and stream processing are two ways of handling data that differ in whether the dataset is bounded and whether results can be delayed. Batch processing collects data into a bounded dataset with a known start and end, then processes the whole chunk in one scheduled run. Because the job sees everything at once, it can do heavy joins and aggregations efficiently and is trivial to re-run. Stream processing handles an unbounded sequence of events continuously, processing each event the moment it arrives, giving very low latency at the cost of more complexity.
The deeper technical difference is how you compute over time. In batch you process a complete, settled chunk, so you rarely worry about ordering. In streaming you never see all the data, so computations rely on windows, and you must handle event-time versus processing-time, late data, and watermarks. Spark Structured Streaming groups events into micro-batches every few hundred milliseconds, whereas Flink processes each event individually for the lowest latency.
The two modes are not rivals: they complement each other. Lambda architecture runs a fast speed layer alongside a complete batch layer, and Kappa simplifies by treating everything as a stream.
In a banking context at the National Bank of India, the fraud team needed immediacy so a streaming job evaluated each transaction the instant it arrived and fired an alert the moment Rajesh's spend crossed a threshold. The finance team needed completeness so a nightly batch job produced correct per-account totals. Streaming answered what is happening right now, and batch answered what happened over the whole day.
Remember one sentence: batch processes a finished pile for completeness and cost, streaming processes each event instantly for freshness, and banks use both.
---