Apache Flink

Flink SQL expresses streaming and batch computations as continuous SQL over dynamic tables backed by Kafka, Hive, JDBC, or CDC, with temporal joins, MATCH_RECOGNIZE, and changelog semantics, and unlike Spark SQL it is true streaming rather than micro-batch.

Stream deduplication keeps only the first or latest event per key, via ROW_NUMBER in Flink SQL or keyed state plus timers in the DataStream API, and it is essential because CDC and event-sourced pipelines can redeliver events.

Kinesis Data Analytics, now Amazon Managed Service for Apache Flink, runs your Flink applications on a fully managed Flink engine, handling provisioning, scaling, checkpointing, and availability.

Managed Flink runs full Apache Flink applications with the complete stateful, event-time feature set for complex production processing, while the legacy SQL-only Kinesis Analytics offers quick streaming SQL with limited windowing and state.

Restart strategies decide whether and how Flink retries a failed job: fixed-delay for constant retries, failure-rate for a tolerance window, exponential-delay for growing backoff, and no-restart to fail fast.

Region failover restarts only the failed pipelined region and its dependents, not the whole job, which is why it is the default and more efficient than full restart.

Flink SQL runs standard SQL over dynamic tables that change as events arrive, compiling the query into a streaming dataflow graph whose results update continuously.

A dynamic table changes over time as rows are inserted, updated, and deleted, whereas a static batch table is a fixed snapshot, which is why queries over it run continuously rather than once.

Append-only emits inserts only, retract emits retraction-plus-new-row pairs for updates, and upsert emits keyed insert-or-update plus delete, and the query's changelog mode must match its sink.

CREATE TABLE ... WITH declares a table's schema and watermark, and the WITH clause binds it to an external system via a connector and a format, so it can act as a source or a sink.

A temporal join enriches each event with the reference-table version valid at that event's time via FOR SYSTEM_TIME AS OF, unlike a stream-stream join that matches two event streams within a window.

MATCH_RECOGNIZE detects sequence patterns in an ordered partitioned stream using regex-like PATTERN and DEFINE clauses, turning complex event processing into declarative SQL.

CDC in Flink SQL turns a database's insert, update, and delete journal into a live changelog table, where each event carries a kind that tells Flink how to update its materialized view.

A Flink SQL catalog is the metadata store for tables and functions; the in-memory one forgets on restart, while HiveCatalog and JdbcCatalog persist and share definitions across jobs and teams.

Flink SQL hints are inline per-query directives; OPTIONS overrides connector options and LOOKUP tunes lookup-join behavior, all without touching the shared DDL.

Flink's runtime is Dispatcher receives jobs and serves the UI, JobManager plans and schedules and checkpoints each job, ResourceManager allocates slots from the cluster manager, and TaskManagers run the work in their slots.

Flink CEP detects ordered, time-bounded event patterns described with a Pattern builder, compiling them into an automaton that fires when a full sequence matches within the window.

Flink CDC uses Debezium-based source connectors to snapshot a database and then tail its transaction log, turning every insert, update, and delete into a real-time changelog stream for ETL and synchronization.

Async I/O issues many external lookups concurrently via an AsyncFunction and unorderedWait or orderedWait so that external latency overlaps instead of blocking, dramatically improving throughput.

Apache Flink is a distributed stream-processing engine that processes each event as it arrives with event-time semantics, keyed state, and exactly-once guarantees.

Use the Flink Web Dashboard for live inspection and export Flink metrics to Prometheus or Grafana, watching checkpoint duration, state size, backpressure, and consumer lag as the key danger signs.

Flink gets exactly-once end to end by pairing checkpointed state with a sink that pre-commits on checkpoint and only commits when the checkpoint is globally confirmed, so failures roll back uncommitted output.

Flink picks an efficient TypeSerializer for primitives, Tuples, Avro, and well-formed POJOs, and falls back to slower Kryo for unanalyzable types, so defining POJOs correctly keeps serialization fast and lets state schema evolve.

Forwarding keeps records on the same partition in order with no shuffle, while redistributing shuffles records across subtasks by a rule like key hashing at a network cost.

Flink backpressure is automatic buffer-based flow control where a slow stage fills its buffers and the pressure propagates upstream to the source, slowing intake with no data loss and no configuration.

Event time is when the event happened, processing time is when the operator handled it, and ingestion time is when it entered Flink; event time is the correct default for auditable results.

Managed Flink removes infrastructure management, auto-scales, integrates with AWS services, and bills pay-per-use, at the cost of less customization and operational lock-in, while the Flink application code stays portable.

Session mode shares one long-lived cluster across jobs, Per-Job mode gives each job its own cluster with main running on the client and is deprecated, and Application mode gives each application its own cluster with main running on the cluster, which is preferred for production.

Late events are dropped by default, kept and recomputed if within allowed lateness, and captured in a side output if you route them there.

A TaskManager's process size splits into JVM heap, managed memory for RocksDB and sorting, network buffers for shuffles, and metaspace plus overhead, tuned by setting process size and adjusting managed and network fractions.

ProcessFunction and KeyedProcessFunction are the low-level API giving per-element access, keyed state, event-time and processing-time timers, and side outputs, used when window operators cannot express the custom logic you need.

DataStream API is low-level and imperative with full control over state, timers, and windows, while Table API and SQL are high-level and declarative with optimizer-planned relational queries; choose based on control versus conciseness.

Parallelism is subtasks per operator and each subtask needs a slot, so parallelism is capped by total slots, and slot sharing lets different operators' subtasks share one slot so a pipeline needs only as many slots as its highest parallelism.