Definition
- Session mode – One long-lived cluster that stays running independently of any job, and you submit multiple jobs to it over time; they share the same JobManager, TaskManagers, and slot pool.
- Per-Job mode – A brand-new dedicated cluster is created for each job and torn down afterward, but the job's
main()runs on the submitting client; YARN-only and now deprecated. - Application mode – A dedicated cluster per application where the application's
main()runs on the cluster (JobManager), so the client just ships the jar and returns immediately; preferred for production on Kubernetes and YARN. - Isolation – Session mode has weak isolation and shared fate; Application and Per-Job modes provide strong isolation with dedicated resources per application.
Imagine the National Bank of India needs meeting rooms for processing tasks. Session mode is one big shared conference hall that stays booked all day where many teams walk in and run their meetings side by side: convenient and quick to start, but teams can disturb each other. Per-Job mode is booking a fresh private room for every single meeting, but the person who books it has to stand at the door doing setup work for the whole time. Application mode is also a fresh private room per project, but now the setup work happens inside the room by the team itself, so the booking clerk at reception is free the moment the room is handed over.
Understanding the Concepts
What is Session mode?
Session mode starts a single Flink cluster that stays running independently of any job. You submit multiple jobs to it over time, and they all share the same JobManager, TaskManagers, and slot pool. The advantage is fast job startup because the cluster is already up.
// Start a session cluster
// $ bin/start-cluster.sh
// Submit jobs to the running cluster
// $ bin/flink run -m jobmanager:8081 \
// target/nbi-fraud-pipeline.jar
// Submit another job to the same cluster
// $ bin/flink run -m jobmanager:8081 \
// target/nbi-settlement-report.jar
Session mode layout: ┌─────────────────────────────────────┐ │ Shared Cluster │ │ JobManager (long-running) │ │ TaskManagers (shared slot pool) │ │ │ │ Job-1: Fraud Pipeline (slots 1-4)│ │ Job-2: Settlement Report (slots 5-6)│ │ Job-3: Dashboard Agg (slots 7-8)│ └─────────────────────────────────────┘
What is Per-Job mode?
Per-Job mode creates a brand-new dedicated cluster for each submitted job and tears it down when the job finishes. Each job gets isolated resources. However, the job's main() method runs on the client that submits the job, so the client builds the JobGraph and must carry the dependencies.
# Per-Job mode on YARN (deprecated)
$ bin/flink run -m yarn-cluster \
-yn 4 \
-ys 4 \
target/nbi-fraud-pipeline.jar
# Client builds JobGraph locally
# Client stays involved until job submission completes
# YARN creates a dedicated cluster for this one job
Per-Job mode (deprecated):
Client machine:
- Runs main()
- Builds JobGraph
- Ships dependencies
- Bottleneck if submitting many jobs
YARN cluster (dedicated):
- JobManager + TaskManagers
- Torn down when job finishes
What is Application mode?
Application mode gives each application its own dedicated cluster and runs the application's main() on the cluster (JobManager) rather than the client. The client just ships the application jar and returns immediately. This is the preferred mode for production.
# Application mode on Kubernetes
$ bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=nbi-fraud \
-Dkubernetes.container.image=nbi/flink:1.18 \
-Dtaskmanager.numberOfTaskSlots=4 \
local:///opt/flink/usrlib/nbi-fraud-pipeline.jar
# Client returns immediately
# main() runs on JobManager inside the cluster
# Cluster lifecycle = application lifecycle
package com.bank.fraud;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
// This main() runs ON THE CLUSTER in Application mode
public class FraudPipeline {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.addSource(new KafkaSource<>("nbi-transactions"))
.keyBy(Transaction::getAccountId)
.process(new FraudRuleEngine())
.addSink(new AlertSink());
env.execute("NBI Fraud Pipeline");
}
}
How is Application mode different from Session and Per-Job modes?
| Feature | Session Mode | Per-Job Mode | Application Mode |
|---|---|---|---|
| Cluster per job | No (shared) | Yes (dedicated) | Yes (dedicated) |
| Where main() runs | Client | Client | Cluster (JobManager) |
| Isolation | Weak (shared fate) | Strong | Strong |
| Job startup | Fast (cluster ready) | Slow (creates cluster) | Moderate (creates cluster) |
| Client involvement | Stays for submission | Builds JobGraph | Ships jar and returns |
| Resource lifecycle | Cluster outlives jobs | Cluster = one job | Cluster = one application |
| Status | Active | Deprecated (YARN only) | Preferred for production |
Session: Client ──submit──▶ Shared Cluster (many jobs) Per-Job: Client ──main()──▶ Dedicated Cluster (one job) Application: Client ──ship jar──▶ Dedicated Cluster ──main()──▶ (one app)
How does Application mode work on Kubernetes?
Native Kubernetes Application mode lets Flink request its own pods for the JobManager and TaskManagers. The application is deployed as a self-contained unit, fitting cloud-native operations and GitOps workflows.
# Kubernetes Application mode deployment
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: nbi-fraud-pipeline
spec:
image: nbi/flink:1.18
flinkVersion: v1_18
flinkConfiguration:
taskmanager.numberOfTaskSlots: "4"
state.backend: rocksdb
state.checkpoints.dir: s3://nbi-checkpoints/fraud/
serviceAccount: flink
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "4096m"
cpu: 2
replicas: 3
job:
jarURI: local:///opt/flink/usrlib/nbi-fraud-pipeline.jar
parallelism: 4
upgradeMode: savepoint
What are common mistakes when choosing deployment modes?
The most common mistake is running production streaming jobs on a shared session cluster where a rogue job can starve or crash the production pipeline. Another mistake is still using Per-Job mode when Application mode is available and provides the same isolation without the client-side bottleneck.
# WRONG: production fraud job on shared session cluster
$ bin/flink run -m session-cluster:8081 target/nbi-fraud.jar
# Risk: another job takes all slots, TM crash affects fraud job
# WRONG: still using deprecated Per-Job mode
$ bin/flink run -m yarn-cluster target/nbi-fraud.jar
# Client builds JobGraph, stays involved, bottleneck
# CORRECT: Application mode on Kubernetes
$ bin/flink run-application --target kubernetes-application \
-Dkubernetes.cluster-id=nbi-fraud \
local:///opt/flink/usrlib/nbi-fraud.jar
# Isolated cluster, main() on cluster, client returns immediately
When should you use each deployment mode?
Use Session mode for development, ad hoc queries, and many short interactive jobs where fast startup and efficient sharing matter more than isolation. Use Application mode for production streaming and batch applications where isolation, no client-side bottleneck, and clean resource lifecycle are essential. Avoid Per-Job mode since Application mode supersedes it.
Decision guide:
Development and exploration? --> Session mode
Ad hoc short-lived queries? --> Session mode
Production streaming job? --> Application mode
Production batch application? --> Application mode
Kubernetes deployment? --> Application mode (native K8s)
YARN legacy deployment? --> Application mode (not Per-Job)
What should a senior engineer know about deployment modes?
A senior engineer should understand that Application mode's key advantage is running main() on the cluster so the client never becomes a bottleneck, that a single application can contain multiple jobs executed in sequence within its dedicated cluster, and that the Flink Kubernetes Operator provides declarative management with savepoint-based upgrades. They should also know that max parallelism must be set before the first savepoint if you plan to scale later.
package com.bank.streaming;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class MultiJobApplication {
// In Application mode, main() runs on the cluster
// A single application can contain multiple sequential jobs
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setMaxParallelism(128); // set before first savepoint
// Job 1: Preprocess
env.addSource(new KafkaSource<>("nbi-raw-transactions"))
.map(Transaction::normalize)
.addSink(new KafkaSink<>("nbi-clean-transactions"));
env.execute("NBI Preprocess Job");
// Job 2: Fraud scoring (same application, same cluster)
env.addSource(new KafkaSource<>("nbi-clean-transactions"))
.keyBy(Transaction::getAccountId)
.process(new FraudRuleEngine())
.addSink(new AlertSink());
env.execute("NBI Fraud Scoring Job");
}
}
Visual Overview
┌────────────────────────────────────────────────────┐ │ Session Mode │ │ │ │ Client ──submit──▶ ┌─────────────────────────┐ │ │ Client ──submit──▶ │ Shared Cluster │ │ │ Client ──submit──▶ │ JobMgr + TaskMgrs │ │ │ │ Job-1, Job-2, Job-3 │ │ │ └─────────────────────────┘ │ │ + Fast startup, shared resources │ │ - Weak isolation, shared fate │ └────────────────────────────────────────────────────┘ ┌────────────────────────────────────────────────────┐ │ Per-Job Mode (Deprecated) │ │ │ │ Client ──main()──▶ ┌───────────────────────┐ │ │ (builds JobGraph) │ Dedicated Cluster │ │ │ │ JobMgr + TaskMgrs │ │ │ │ One job only │ │ │ └───────────────────────┘ │ │ + Isolation │ │ - Client bottleneck, YARN only │ └────────────────────────────────────────────────────┘ ┌────────────────────────────────────────────────────┐ │ Application Mode (Preferred) │ │ │ │ Client ──ship jar──▶ ┌───────────────────────┐ │ │ (returns immediately) │ Dedicated Cluster │ │ │ │ JobMgr runs main() │ │ │ │ TaskMgrs for this app │ │ │ └───────────────────────┘ │ │ + Isolation, no client bottleneck │ │ + Clean lifecycle, K8s native support │ └────────────────────────────────────────────────────┘
Key Takeaway
Flink has three deployment modes that differ in how jobs map to clusters and where the job's main() method runs. Session mode runs one long-lived cluster that many jobs submit to and share, so they all use the same JobManager, TaskManagers, and slot pool. Its advantages are fast job startup and efficient sharing for many small or short jobs, which makes it ideal for development and interactive exploration. Its weakness is isolation: jobs compete for the same slots, one job can starve others, and a TaskManager failure can hit multiple jobs.
Per-Job mode creates a dedicated cluster for each job but runs main() on the client, so the client builds the JobGraph and must carry the dependencies, creating a bottleneck. It is YARN-only and now deprecated. Application mode gives each application its own dedicated cluster and runs main() on the JobManager, so the client just ships the jar and returns immediately. This is the preferred mode for production because it combines strong isolation, no client-side bottleneck, and a clean resource lifecycle.
At the National Bank of India we learned this painfully because we ran everything on one shared session cluster. A heavy experimental job once took the slots the fraud job needed, and when an experiment crashed a TaskManager it disrupted the production job. We moved the critical fraud job to Application mode on Kubernetes, which gave it its own isolated cluster with main() running on the JobManager. We kept a session cluster for developers' interactive work.
That split, Session mode for development and Application mode for production, is the practical rule that every team should follow.
Remember one sentence: Application mode gives each job its own isolated cluster with main() running on the JobManager so the client returns immediately and there is no shared-fate risk.
---