Back to Apache Flink
Q20Question 20 of 33

What is Apache Flink

Key Concepts
  • Flink is an open-source distributed stream-processing engine that processes each event as it arrives with sub-second latency.
  • It distinguishes event time from processing time and uses watermarks to handle out-of-order and late data correctly.
  • Keyed state is first-class and backed by configurable state backends such as HashMapStateBackend or RocksDB for large state.
  • Distributed checkpointing with lightweight barriers gives exactly-once state consistency and, with transactional sinks, end-to-end exactly-once.
  • Flink treats batch as a special case of streaming over a bounded dataset so one API serves both.
  • The DataStream API gives low-level control while the Table API and Flink SQL provide declarative relational queries with optimizer support.
  • Flink is the engine of choice for real-time fraud scoring, live risk limits, and instant settlement signals where correctness and latency both matter.

Definition

Apache Flink is a distributed stream-processing engine that processes each event as it arrives with event-time semantics, keyed state, and exactly-once guarantees.
  • Stream-first engine – Flink treats a stream as the fundamental primitive and processes each record the instant it arrives, unlike micro-batch systems that group events into small batches.
  • Event-time processing – It distinguishes event time (when it happened) from processing time (when the engine sees it) and uses watermarks to handle out-of-order data.
  • Stateful processing – Keyed state is first-class, backed by configurable state backends such as HashMapStateBackend for fast in-memory access or RocksDBStateBackend for large state that exceeds heap.
  • Exactly-once guarantees – Distributed checkpointing with lightweight barriers snapshots all operator state and source offsets, giving exactly-once state consistency and, with transactional sinks, end-to-end exactly-once.
  • Unified batch and streaming – Flink treats batch as a special, bounded case of streaming so the same API serves both use cases.

Think of the National Bank of India's fraud desk. Spark's default mode is like a supervisor who collects card swipes into small stacks and checks each stack every few seconds. Flink is like a sharp officer who inspects every single swipe the instant it lands on the counter, remembers each customer's recent pattern in a personal ledger (state), and can tell that Priya's card was used in two cities one minute apart right now, not a few seconds late. Flink also keeps a reliable notebook (checkpoint) so if the officer faints, a replacement resumes from the exact last page with no swipe double-counted.

Understanding the Concepts

Apache Flink is an open-source, distributed stream-processing engine designed for stateful computations over unbounded and bounded data streams. Its core model is the dataflow graph: your program is a directed acyclic graph of operators (sources, transformations, sinks) through which events flow continuously. Every event is processed as it arrives, so latency can be in the low milliseconds.

JAVA
package com.bank.streaming;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;

public class FlinkIntroDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        DataStream<String> transactions = env.socketTextStream("localhost", 9092);
        transactions
            .filter(txn -> txn.contains("FRAUD"))
            .print("Suspicious");

        env.execute("NBI Fraud Alert Demo");
    }
}
TEXT
Suspicious> FRAUD:ACC-10045:Rs5000:Mumbai
Suspicious> FRAUD:ACC-10045:Rs8000:Delhi

Before Flink, real-time stream processing relied on micro-batch engines like Spark Structured Streaming or on low-level frameworks like Storm that lacked state management. Flink was created to give developers a true event-at-a-time engine with built-in state, event-time semantics, and exactly-once guarantees. For a bank, this means fraud detection, live risk limits, and instant settlement signals can run with sub-second latency and correct results even when events arrive late or out of order.

JAVA
package com.bank.streaming;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;

public class WhyFlinkDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        // Event-time window: correct even if events arrive late
        env.fromElements(
                new Transaction("ACC-001", 5000.00, 1700000000000L),
                new Transaction("ACC-001", 3000.00, 1700000010000L))
            .assignTimestampsAndWatermarks(
                WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(
                    Duration.ofSeconds(5))
                .withTimestampAssigner((txn, ts) -> txn.getEventTime()))
            .keyBy(Transaction::getAccountId)
            .window(TumblingEventTimeWindows.of(Time.minutes(1)))
            .sum("amount")
            .print("Per-minute total");

        env.execute("NBI Event-Time Demo");
    }
}

Flink's runtime is built on a pipelined dataflow execution model. The JobManager splits the user program into an execution graph of parallel subtasks. Each subtask runs inside a task slot on a TaskManager. Records flow through the graph continuously via network buffers. When a shuffle is needed (for example after a keyBy), records are serialized, sent over the network to the correct downstream subtask, and deserialized. When operators can be chained (same parallelism, no shuffle), Flink fuses them into a single thread, eliminating serialization overhead.

JAVA
package com.bank.streaming;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class InternalProcessingDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(4); // 4 subtasks per operator

        env.fromElements(
                "ACC-001:5000", "ACC-002:3000", "ACC-001:2000", "ACC-003:7000")
            .map(line -> {
                String[] parts = line.split(":");
                return new Transaction(parts[0],
                    Double.parseDouble(parts[1]), System.currentTimeMillis());
            })
            .keyBy(Transaction::getAccountId) // shuffle by account
            .sum("amount")
            .print("Running total");

        env.execute("NBI Internal Processing Demo");
    }
}
TEXT
Running total> Transaction{accountId=ACC-001, amount=5000.0}
Running total> Transaction{accountId=ACC-002, amount=3000.0}
Running total> Transaction{accountId=ACC-001, amount=7000.0}
Running total> Transaction{accountId=ACC-003, amount=7000.0}

| Feature | Apache Flink | Spark Structured Streaming |
|---|---|---|
| Processing model | True event-at-a-time | Micro-batch (continuous mode experimental) |
| Latency | Low milliseconds | Hundreds of milliseconds to seconds |
| State management | First-class keyed state with RocksDB | State store with watermark, less flexible |
| Event-time handling | Native watermarks, allowed lateness, side outputs | Watermark support, less granular late handling |
| Exactly-once | Barrier-based checkpoints + 2PC sinks | Micro-batch transaction per batch |
| Batch mode | Bounded stream (special case) | Native batch engine |
| API | DataStream, Table API, Flink SQL | DataFrame, Structured Streaming API |

JAVA
// Flink: true event-at-a-time with sub-second latency
package com.bank.streaming;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class FlinkVsSparkDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        // Each transaction processed instantly, no micro-batch delay
        env.addSource(new KafkaSource<>("nbi-transactions"))
            .keyBy(Transaction::getAccountId)
            .process(new FraudScoringFunction()) // per-event, stateful
            .addSink(new AlertSink());

        env.execute("NBI Real-Time Fraud Scoring");
    }
}

A Flink cluster has two main processes. The JobManager is the coordinator: it receives the job graph, schedules tasks across TaskManagers, triggers checkpoints, and handles failover. The TaskManager is the worker: it runs the actual subtasks inside task slots, manages local state, and exchanges data over the network. A ResourceManager handles resource allocation depending on the deployment mode (YARN, Kubernetes, or standalone).

JAVA
// Configuration example for a Flink cluster
// flink-conf.yaml
// jobmanager.rpc.address: jobmanager-host
// taskmanager.numberOfTaskSlots: 4
// taskmanager.memory.process.size: 4096m
// state.backend: rocksdb
// state.checkpoints.dir: s3://nbi-checkpoints/flink/
TEXT
Cluster layout:
  JobManager (1 instance)
    - Receives job graph
    - Schedules subtasks
    - Triggers checkpoints

  TaskManager (N instances, each with M slots)
    - Runs subtasks
    - Manages local state
    - Exchanges data via network buffers

Flink provides two production state backends. The HashMapStateBackend keeps state as Java objects on the JVM heap, giving fast access but limited by heap size. The EmbeddedRocksDBStateBackend stores state in RocksDB on local disk with a block cache in managed off-heap memory, supporting state much larger than memory at the cost of serialization overhead on every access. Both backends checkpoint to a durable store such as S3 or HDFS.

JAVA
package com.bank.streaming;

import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class StateBackendDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        // Use RocksDB for large per-account state
        env.setStateBackend(new EmbeddedRocksDBStateBackend(true));
        env.getCheckpointConfig().setCheckpointStorage(
            "s3://nbi-checkpoints/flink/");

        env.addSource(new KafkaSource<>("nbi-transactions"))
            .keyBy(Transaction::getAccountId)
            .process(new AccountAggregator()) // months of per-account state
            .addSink(new LedgerSink());

        env.execute("NBI Stateful Aggregation");
    }
}

Flink enforces several rules that developers must understand. Parallelism cannot exceed the total number of task slots available. Keyed state is scoped to the current key and cannot be accessed across keys. Watermarks must advance monotonically within a source partition. Checkpoints require replayable sources (like Kafka) for exactly-once recovery. State classes used with the RocksDB backend must be serializable by Flink's type system.

JAVA
package com.bank.streaming;

import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.util.Collector;
import java.math.BigDecimal;

public class StateRulesDemo
        extends KeyedProcessFunction<String, Transaction, String> {

    // State is scoped to the current key (account)
    private transient ValueState<BigDecimal> runningTotal;

    @Override
    public void open(Configuration parameters) {
        runningTotal = getRuntimeContext().getState(
            new ValueStateDescriptor<>("runningTotal", BigDecimal.class));
    }

    @Override
    public void processElement(Transaction txn, Context ctx,
            Collector<String> out) throws Exception {
        BigDecimal current = runningTotal.value();
        if (current == null) current = BigDecimal.ZERO;
        BigDecimal updated = current.add(txn.getAmount());
        runningTotal.update(updated);
        out.collect("Account " + ctx.getCurrentKey()
            + " running total: Rs" + updated);
    }
}

The most common mistake is using processing time when event time is needed, which gives non-deterministic results that break reconciliation. Another mistake is not configuring a state backend for production, leaving the default heap backend that cannot handle large state. Forgetting to set checkpoint storage means no recovery on failure. Using blocking I/O calls inside operators kills throughput.

JAVA
// WRONG: using processing time for financial aggregation
stream
    .keyBy(Transaction::getAccountId)
    .window(TumblingProcessingTimeWindows.of(Time.minutes(1)))
    .sum("amount"); // non-deterministic: replay gives different results

// CORRECT: using event time with watermarks
stream
    .assignTimestampsAndWatermarks(
        WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(
            Duration.ofSeconds(5))
        .withTimestampAssigner((txn, ts) -> txn.getEventTime()))
    .keyBy(Transaction::getAccountId)
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))
    .sum("amount"); // deterministic: same data always gives same result

Use Flink when you need real-time processing with sub-second latency, stateful computations over streams (fraud detection, running totals, session analysis), event-time correctness with out-of-order data, and exactly-once guarantees. Banking use cases include real-time fraud scoring, live risk limit enforcement, instant settlement signals, continuous regulatory reporting, and real-time customer dashboards.

JAVA
package com.bank.fraud;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class FraudPipelineDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();
        env.enableCheckpointing(60_000); // checkpoint every minute

        env.addSource(new KafkaSource<>("nbi-card-swipes"))
            .assignTimestampsAndWatermarks(nbiWatermarkStrategy())
            .keyBy(CardSwipe::getAccountId)
            .process(new FraudRuleEngine()) // stateful, timer-driven
            .addSink(new AlertService());

        env.execute("NBI Real-Time Fraud Pipeline");
    }
}

Flink is not the right choice for simple batch ETL where latency does not matter and Spark or a warehouse query engine is simpler. It is overkill for low-volume event processing where a serverless function (AWS Lambda) suffices. It adds unnecessary complexity for request-response microservice patterns where a REST API is the natural fit.

TEXT
Decision guide:
  Need sub-second latency?           --> YES: consider Flink
  Need stateful stream processing?    --> YES: Flink is ideal
  Need event-time correctness?        --> YES: Flink with watermarks
  Simple daily batch ETL?             --> NO: use Spark or warehouse SQL
  Low volume, few events per second?  --> NO: use Lambda or a simple consumer
  Request-response pattern?           --> NO: use a REST microservice

A senior engineer should understand the interaction between checkpointing interval and end-to-end latency, how RocksDB state backend performance depends on managed memory sizing, the difference between aligned and unaligned checkpoints under backpressure, savepoint compatibility and state schema evolution, and how to monitor a production job using Flink metrics exported to Prometheus and Grafana.

JAVA
package com.bank.streaming;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.environment.CheckpointConfig;

public class SeniorConfigDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        // Senior-level checkpoint tuning
        env.enableCheckpointing(30_000); // 30s interval
        CheckpointConfig cfg = env.getCheckpointConfig();
        cfg.setMinPauseBetweenCheckpoints(10_000);
        cfg.setCheckpointTimeout(120_000);
        cfg.enableUnalignedCheckpoints(); // for backpressure tolerance
        cfg.setExternalizedCheckpointCleanup(
            CheckpointConfig.ExternalizedCheckpointCleanup
                .RETAIN_ON_CANCELLATION);

        env.execute("NBI Production Config Demo");
    }
}

Visual Overview

DIAGRAM
                        Apache Flink Architecture

  ┌──────────────────────────────────────────────────────────────┐
  │                        JobManager                            │
  │  ┌──────────┐  ┌──────────────┐  ┌───────────────────────┐  │
  │  │Dispatcher│  │ResourceMgr   │  │   CheckpointCoord     │  │
  │  └──────────┘  └──────────────┘  └───────────────────────┘  │
  └──────────────────────┬───────────────────────────────────────┘
                         │ schedules tasks
           ┌─────────────┼─────────────┐
           ▼             ▼             ▼
  ┌────────────┐ ┌────────────┐ ┌────────────┐
  │TaskManager │ │TaskManager │ │TaskManager │
  │ ┌───┐┌───┐│ │ ┌───┐┌───┐│ │ ┌───┐┌───┐│
  │ │S1 ││S2 ││ │ │S3 ││S4 ││ │ │S5 ││S6 ││
  │ └───┘└───┘│ │ └───┘└───┘│ │ └───┘└───┘│
  └────────────┘ └────────────┘ └────────────┘
      S = Task Slot (runs subtask pipeline)


  ┌─────────────────────────────────────────────┐
  │          Flink Dataflow Execution           │
  │                                             │
  │  Kafka ──▶ Source ──▶ Map ──▶ KeyBy         │
  │                               (shuffle)     │
  │              ┌───────────┬──────────┐        │
  │              ▼           ▼          ▼        │
  │           Subtask-0  Subtask-1  Subtask-2   │
  │           (key=ACC1) (key=ACC2) (key=ACC3)  │
  │              │           │          │        │
  │              ▼           ▼          ▼        │
  │            Sink        Sink       Sink       │
  └─────────────────────────────────────────────┘


  ┌─────────────────────────────────────────────┐
  │       Checkpoint Barrier Flow               │
  │                                             │
  │  Source ──barrier──▶ Map ──barrier──▶ Sink  │
  │    │                  │                │    │
  │    ▼                  ▼                ▼    │
  │  snapshot           snapshot        snapshot│
  │  (offset)           (state)        (pending)│
  │    │                  │                │    │
  │    └──────────────────┴────────────────┘    │
  │                   │                         │
  │                   ▼                         │
  │          Durable Storage (S3/HDFS)          │
  └─────────────────────────────────────────────┘

Key Takeaway

Apache Flink is an open-source distributed stream-processing engine built for true record-at-a-time processing with very low latency and strong correctness guarantees. Its signature difference from a micro-batch system like Spark Structured Streaming is that Flink treats a stream as the fundamental primitive and processes each event the instant it arrives, rather than grouping events into small batches, so latency can be in the low milliseconds. Flink even treats batch as a special case of streaming over a bounded dataset, so one API serves both.

Flink's standout features are time handling, state, and correctness. For time, it distinguishes event time from processing time and uses watermarks to decide when a window is complete, with configurable allowed lateness for stragglers. For state, keyed state is first-class: each key, such as a bank account, can keep running state like a count or last-seen location, managed in a state backend that can be in-memory or RocksDB for very large state, which makes stateful fraud rules natural and scalable. For correctness, Flink takes periodic distributed checkpoints of all operator state and source offsets using lightweight barriers, so after a failure it restores the latest checkpoint and replays from saved offsets, giving exactly-once state semantics, and with transactional sinks that extends to exactly-once end to end.

At the National Bank of India this was the engine we reached for whenever real time and correctness both mattered, like real-time fraud scoring. The job keyed the swipe stream by account, kept a per-account count inside a one-minute event-time window, and fired an alert the instant an account exceeded its allowed swipes. Because state was checkpointed, even a mid-stream crash and restore did not lose the running counts or double-count a swipe. The DataStream API gave us full control for the custom fraud rules, while the Table API and Flink SQL served the reporting aggregations that analysts maintained.

That combination of event-at-a-time latency, rich keyed state, event-time windows, and exactly-once guarantees is exactly why a bank chooses Flink for its most demanding streaming workloads.

Remember one sentence: Flink is a true event-at-a-time streaming engine with event-time watermarks, keyed state, and exactly-once checkpointing that processes each record the instant it arrives.

---

Share
Use ← → arrow keys to navigate