Definition
- Dispatcher – The entry point that receives job submissions over REST, serves the web dashboard, and spawns a dedicated JobManager for each job.
- JobManager – The master of a single job that converts the logical JobGraph into a physical ExecutionGraph, schedules subtasks onto slots, and runs the checkpoint coordinator.
- ResourceManager – Acquires and manages execution slots on TaskManagers, pluggable to standalone, YARN, or Kubernetes backends.
- TaskManager – A worker process whose slots actually run subtask threads, hold operator state, and exchange data through network buffers.
- Slot – The unit of resource isolation on a TaskManager, representing a fixed slice of memory and one thread of execution.
Imagine the National Bank of India runs a large back-office processing operation. The Dispatcher is the front desk that receives new work orders and the public status board. For each work order it appoints a project manager, the JobManager, who turns the high-level plan into detailed steps, assigns them to staff, and keeps the master checkpoint so that if anything crashes the work resumes from the last safe point. The ResourceManager is the staffing coordinator who finds free desks for the workers. The TaskManagers are the clerks sitting at desks, and each desk is one slot where the real processing happens.
Understanding the Concepts
What is the Dispatcher?
The Dispatcher is the long-lived entry point of a Flink cluster. It exposes the REST endpoint that the command-line client and other tools use to submit jobs, and it serves the web dashboard where you watch running jobs, metrics, and checkpoints. For each submitted job, the Dispatcher starts a dedicated JobManager.
package com.bank.flink;
import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
public class SubmitToDispatcher {
public static void main(String[] args) throws Exception {
Configuration config = new Configuration();
config.setString(RestOptions.ADDRESS, "flink-dispatcher.bank.internal");
config.setInteger(RestOptions.PORT, 8081);
// The Dispatcher receives this job over REST
// and spawns a dedicated JobManager for it
RestClusterClient<?> client = new RestClusterClient<>(config, "nbi-fraud-job");
System.out.println("Connected to Dispatcher at " + config.getString(RestOptions.ADDRESS, ""));
}
}
What is the JobManager?
The JobManager is the master of a single job. It receives the JobGraph, which is the logical dataflow of operators, and transforms it into the ExecutionGraph, the physical plan that accounts for parallelism. A source operator with parallelism four becomes four parallel subtasks. The JobManager schedules these subtasks onto available slots, tracks their status, and triggers periodic checkpoints.
package com.bank.flink;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;
public class JobGraphDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(4); // JobManager creates 4 subtasks per operator
DataStream<String> transactions = env
.socketTextStream("kafka-broker.bank.internal", 9092);
transactions
.filter(txn -> txn.contains("FRAUD"))
.print();
// When submitted, JobManager converts this to an ExecutionGraph:
// Source(4 subtasks) -> Filter(4 subtasks) -> Sink(4 subtasks)
env.execute("NBI-Fraud-Detection");
}
}
JobGraph (logical): Source -> Filter -> Sink
ExecutionGraph: Source[0..3] -> Filter[0..3] -> Sink[0..3]
Total subtasks: 12 (4 per operator)
What is the ResourceManager?
The ResourceManager acquires and manages TaskManager slots. It is pluggable to different deployment backends. In standalone mode, TaskManagers are already running and register their slots. Under YARN or Kubernetes, the ResourceManager dynamically requests new TaskManager containers when a job needs more slots and releases them when idle.
// Flink configuration for Kubernetes ResourceManager
// flink-conf.yaml
// kubernetes.cluster-id: nbi-flink-cluster
// kubernetes.namespace: bank-streaming
// taskmanager.numberOfTaskSlots: 4
// taskmanager.memory.process.size: 4096m
// The ResourceManager requests pods from Kubernetes
// when the JobManager needs slots for a new job.
Resource allocation flow:
1. JobManager requests 12 slots for the fraud job
2. ResourceManager checks existing TaskManagers
3. If not enough free slots, requests new pods from Kubernetes
4. New TaskManagers start, register slots with ResourceManager
5. ResourceManager offers slots to JobManager
6. JobManager deploys subtasks into the slots
What are TaskManagers and slots?
TaskManagers are the worker processes. Each runs one or more task slots, and each slot executes the subtasks assigned to it as threads, holds the operator state, and manages network buffers for data exchange. A slot is a fixed slice of the TaskManager's memory.
package com.bank.flink;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.TaskManagerOptions;
public class TaskManagerConfig {
public static void main(String[] args) {
Configuration config = new Configuration();
config.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 4);
// Each TaskManager has 4 slots
// Each slot gets 1/4 of the TaskManager's managed memory
// Slot sharing allows subtasks of different operators in the same slot
System.out.println("Slots per TaskManager: "
+ config.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, 1));
}
}
How does a job flow through the architecture?
The complete flow from submission to execution follows a clear sequence.
- You package your Flink application as a JAR and submit it to the Dispatcher over REST.
- The Dispatcher starts a dedicated JobManager for this job.
- The JobManager builds the ExecutionGraph from the JobGraph, computing parallel subtasks.
- The JobManager asks the ResourceManager for the required number of slots.
- The ResourceManager allocates slots on existing TaskManagers or spins up new ones.
- TaskManagers register their offered slots with the ResourceManager.
- The JobManager deploys each subtask into a slot.
- TaskManagers run the subtasks and stream data between them.
- The JobManager periodically triggers checkpoints.
- If a TaskManager fails, the JobManager detects it, restores from the last checkpoint, and reschedules.
package com.bank.flink;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.CheckpointingMode;
public class FlinkJobFlowDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(4);
env.enableCheckpointing(30000, CheckpointingMode.EXACTLY_ONCE);
// Step 1-2: Submit to Dispatcher, which spawns JobManager
// Step 3: JobManager builds ExecutionGraph
// Step 4-6: ResourceManager allocates slots
// Step 7-8: Subtasks deployed and running
// Step 9: Checkpoint every 30 seconds
env.fromElements("txn-001", "txn-002", "txn-003")
.map(txn -> "Processed: " + txn)
.print();
env.execute("NBI-Job-Flow-Demo");
}
}
What is slot sharing and why does it matter?
Slot sharing allows subtasks of different operators in the same job to share a single slot. Without slot sharing, a job with three operators at parallelism four would need 12 slots. With slot sharing, each slot can run one subtask from each operator, so the job needs only 4 slots.
Without slot sharing (12 slots needed):
Slot 1: Source[0] Slot 5: Filter[0] Slot 9: Sink[0]
Slot 2: Source[1] Slot 6: Filter[1] Slot 10: Sink[1]
Slot 3: Source[2] Slot 7: Filter[2] Slot 11: Sink[2]
Slot 4: Source[3] Slot 8: Filter[3] Slot 12: Sink[3]
With slot sharing (4 slots needed):
Slot 1: Source[0] + Filter[0] + Sink[0]
Slot 2: Source[1] + Filter[1] + Sink[1]
Slot 3: Source[2] + Filter[2] + Sink[2]
Slot 4: Source[3] + Filter[3] + Sink[3]
What is high availability for the JobManager?
For long-running streaming jobs, a JobManager failure would be catastrophic without high availability. Flink stores the JobManager metadata in a highly available service like ZooKeeper or Kubernetes, so a standby JobManager can take over if the leader fails.
// flink-conf.yaml for HA with ZooKeeper
// high-availability: zookeeper
// high-availability.zookeeper.quorum: zk1.bank.internal:2181,zk2.bank.internal:2181
// high-availability.storageDir: hdfs:///flink/ha
// high-availability.zookeeper.path.root: /flink-nbi
// The active JobManager stores its metadata in ZooKeeper.
// If it dies, a standby detects the leadership loss,
// recovers the metadata, and resumes the job from the last checkpoint.
What should a senior engineer know?
A senior engineer should be able to trace a job through all four components and explain how each contributes to fault tolerance.
- The Dispatcher is long-lived while JobManagers are per-job, so a Dispatcher failure does not kill running jobs.
- The JobManager contains the scheduler and the checkpoint coordinator, which are the two most critical pieces for correctness.
- Under Kubernetes, the ResourceManager uses pod templates to request TaskManagers with specific resource profiles.
- Slot sharing reduces the total slots needed but means that a slow operator in a shared slot can affect the throughput of faster operators.
- The checkpoint coordinator in the JobManager sends checkpoint barriers through the data channels, and each TaskManager snapshots its state when it receives the barrier.
// Senior tip: check the Flink dashboard for architecture health
// http://flink-dispatcher:8081/#/overview
// Shows: running jobs, TaskManagers, total slots, available slots
// http://flink-dispatcher:8081/#/job/<job-id>/checkpoints
// Shows: checkpoint duration, size, alignment time
Visual Overview
Flink Architecture – Job Submission Flow
┌─────────────┐
│ Client │
│ (flink run) │
└──────┬──────┘
│ REST submit JAR
▼
┌─────────────────┐
│ Dispatcher │
│ (REST + Web UI) │
└──────┬──────────┘
│ spawns per job
▼
┌─────────────────────────────────┐
│ JobManager │
│ ┌───────────┐ ┌──────────────┐ │
│ │ Scheduler │ │ Checkpoint │ │
│ │ │ │ Coordinator │ │
│ └───────────┘ └──────────────┘ │
│ JobGraph → ExecutionGraph │
└──────┬──────────────────────────┘
│ requests slots
▼
┌──────────────────┐
│ ResourceManager │
│ (YARN / K8s / │
│ Standalone) │
└──────┬───────────┘
│ allocates slots
▼
┌──────────────────────────────────┐
│ TaskManagers │
│ ┌──────┐ ┌──────┐ ┌──────┐ │
│ │Slot 1│ │Slot 2│ │Slot 3│ ... │
│ │ task │ │ task │ │ task │ │
│ └──────┘ └──────┘ └──────┘ │
│ Network buffers for data exchange│
└──────────────────────────────────┘
Slot Sharing Example TaskManager 1 TaskManager 2 ┌────────────────────┐ ┌────────────────────┐ │ Slot 1 │ │ Slot 3 │ │ Source[0] │ │ Source[2] │ │ Filter[0] │ │ Filter[2] │ │ Sink[0] │ │ Sink[2] │ ├────────────────────┤ ├────────────────────┤ │ Slot 2 │ │ Slot 4 │ │ Source[1] │ │ Source[3] │ │ Filter[1] │ │ Filter[3] │ │ Sink[1] │ │ Sink[3] │ └────────────────────┘ └────────────────────┘
Key Takeaway
Flink's runtime is a master-worker architecture with four components. The Dispatcher is the long-lived entry point that receives job submissions over REST, serves the web dashboard, and spawns a dedicated JobManager for each job. The JobManager is the brain of a single job: it converts the logical JobGraph into the physical ExecutionGraph, schedules subtasks onto slots, and runs the checkpoint coordinator that periodically snapshots all operator state to durable storage.
The ResourceManager acquires and manages execution slots and is pluggable to the deployment backend, so under YARN or Kubernetes it dynamically requests new TaskManager containers when more slots are needed. The TaskManagers are the worker processes whose slots actually run the subtask threads, hold operator state, and exchange data streams through network buffers. Slot sharing lets subtasks of different operators share a slot to reduce the total slots a job needs.
At the National Bank of India we ran a continuous fraud-scoring job on Flink, and understanding this architecture was exactly what let us size the cluster and reason about recovery. When a TaskManager crashed, the JobManager detected the failure and restored the job from the last checkpoint rather than restarting from zero, so our fraud state survived the crash. The high-availability setup stored JobManager metadata in ZooKeeper so a standby could take over if the leader died, which was critical for a 24/7 streaming job.
Remember one sentence: Dispatcher receives jobs, JobManager plans and checkpoints them, ResourceManager finds slots, and TaskManagers execute the work in those slots.
---