Spark Streaming & Structured Streaming

Structured Streaming treats a live data stream as an unbounded table processed incrementally with the Catalyst-optimized DataFrame API, event-time support, and exactly-once guarantees, replacing the legacy RDD-based DStream API.

Append writes only new final rows, Complete rewrites the whole result table each trigger for aggregations, and Update writes only the rows that changed, so you pick the mode based on your query type and sink capability.

Event time is the timestamp embedded in the data marking when an event actually happened, processing time is the wall clock when Spark handles it, and you must bucket by event time so late and out-of-order events land in the correct window.

A watermark is max-event-time-seen minus an allowed-lateness delay; events older than it are dropped and closed windows' state is cleaned up, which is what bounds state in stateful streaming.

Stateless operations (filter, map, select) treat each batch independently with no memory, while stateful operations (aggregations, joins, deduplication) carry state across batches and need a state store, watermarks, and checkpointing.

The checkpoint persists read offsets, committed offsets, state, and metadata to reliable storage so a streaming query resumes exactly-once after a failure, but incompatible query changes can make an existing checkpoint unusable.

Use `readStream.format("kafka")` to consume a topic as an unbounded DataFrame and `writeStream.format("kafka")` to produce back; checkpointed offsets give exactly-once on the source side, and idempotent or transactional sinks give exactly-once end-to-end.

ProcessingTime runs batches at a fixed interval, Once runs one batch then stops, AvailableNow drains all data in rate-limited batches then stops, Continuous is experimental low-latency, and maxOffsetsPerTrigger or maxFilesPerTrigger cap how much each batch ingests.

Flink SQL windows a stream with TUMBLE, HOP, SESSION, or CUMULATE via a window table-valued function in the FROM clause, then aggregates by GROUP BY window_start and window_end.

A stream-stream join without a time bound keeps state forever, so you add an interval constraint on event-time columns to let watermarks evict old rows and keep state finite.

Late events that arrive after the watermark has passed the window end are dropped by default in Flink SQL; allowed lateness keeps the window open for a grace period, and DataStream side outputs capture them for separate handling.

A custom watermark strategy pairs a timestamp extractor with a watermark generator that decides when and how far to advance the watermark, and the out-of-orderness setting is the dial between waiting too long and dropping too much.

Checkpoint barriers flow from sources through the dataflow graph, each operator aligns barriers across its inputs then snapshots its state asynchronously to durable storage, producing one globally consistent recovery point.

Keyed state is per-key memory accessed inside a keyed stream for per-entity logic, while operator state is per-parallel-instance memory used by sources, sinks, and broadcasts for per-worker plumbing.

HashMapStateBackend is heap-fast but memory-bounded, while EmbeddedRocksDBStateBackend is disk-backed and handles huge state but is slower per access, so you pick based on state size versus latency needs.

Window functions compute across related rows without collapsing them, so `AVG ... OVER` with `LAG` finds month-over-month growth and `ROW_NUMBER() OVER (PARTITION BY ...)` flags duplicates.

Broadcast state replicates a small, slowly-changing rules or config dataset to every parallel operator instance, where the main stream reads it to drive processing without restarting the job.

A Flink window slices the endless stream into finite buckets for aggregation, and a watermark is the clock that tells each bucket when it is safe to close and emit its result.

The assigner puts events into windows, the trigger decides when each window fires, and the optional evictor trims elements around the firing, and they interact in a strict assign-then-trigger-then-evict order.

Checkpoints are Flink's automatic, cheap, self-owned snapshots for crash recovery, while savepoints are user-triggered, portable, durable snapshots for upgrades, rescaling, and planned changes.

Aligned checkpoints wait for all barriers and exclude in-flight data but stall under backpressure, while unaligned checkpoints snapshot immediately including in-flight data to stay fast under backpressure.

Eager watermarks fire fast but drop more stragglers, relaxed watermarks wait longer but capture them, and you choose based on whether speed or completeness matters more for the use case.