Definition
Databricks offers three main compute types for different workloads.
- All-Purpose clusters – Long-lived, shared, interactive clusters for development and exploration in notebooks. Multiple users attach and run commands interactively.
- Job clusters – Ephemeral clusters created by the Jobs scheduler just before a job runs and terminated immediately after. Each job gets isolated, cost-efficient compute.
- SQL Warehouses – Compute endpoints optimized for SQL analytics and BI, running the Photon engine. Available in classic (your cloud account) and serverless (managed pool, instant start).
The rule is: development on All-Purpose, production on Job clusters, SQL analytics on SQL Warehouses.
Imagine the National Bank of India has three kinds of work vehicles. An All-Purpose cluster is the staff pool car that sits in the lot all day, shared by several employees: convenient for ad hoc trips but costs money while idling. A Job cluster is a delivery van hired for exactly one scheduled delivery, returned the moment it is done, so you pay only for the trip. A SQL Warehouse is a specialized cash-counting machine tuned for one job: counting money fast. The serverless version is already warmed up and ready the instant you flip the switch.
Understanding the Concepts
What is an All-Purpose cluster?
An All-Purpose cluster (formerly called an interactive cluster) is a cluster you start and keep running so that people can attach notebooks and run commands interactively. It is shared, meaning several users work on it at once, and it stays alive until stopped or auto-terminated after an idle period.
# Databricks notebook on an All-Purpose cluster
# Multiple data engineers develop simultaneously
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Interactive exploration of bank transactions
txns = spark.read.format("delta").load("/data/bank/transactions")
txns.filter("branch_code = 'MUMBAI-01'").groupBy("txn_date").count().show()
# Quick debugging and prototyping
accounts = spark.read.format("delta").load("/data/bank/accounts")
accounts.describe("balance").show()
// All-Purpose cluster configuration
{
"cluster_name": "bank-dev-cluster",
"spark_version": "14.3.x-scala2.12",
"node_type_id": "i3.xlarge",
"num_workers": 4,
"autotermination_minutes": 30,
"spark_conf": {
"spark.sql.adaptive.enabled": "true"
}
}
What is a Job cluster?
A Job cluster is created automatically by the Databricks Jobs scheduler at the moment a scheduled job starts, runs only that job, and is terminated the instant the job finishes. Each job run gets its own fresh, isolated cluster.
// Job cluster configuration inside a Databricks Job definition
{
"job_name": "nightly-etl-bank-transactions",
"tasks": [{
"task_key": "load_transactions",
"notebook_task": {
"notebook_path": "/pipelines/transaction_etl"
},
"new_cluster": {
"spark_version": "14.3.x-scala2.12",
"node_type_id": "i3.xlarge",
"num_workers": 8,
"spark_conf": {
"spark.sql.adaptive.enabled": "true",
"spark.sql.shuffle.partitions": "200"
}
}
}],
"schedule": {
"quartz_cron_expression": "0 0 2 * * ?",
"timezone_id": "Asia/Kolkata"
}
}
Job cluster lifecycle: 2:00 AM Job triggers → Cluster CREATED (2 min startup) 2:02 AM ETL notebook starts executing 2:45 AM ETL completes → Cluster TERMINATED Cost: 43 minutes of compute, zero idle time
What is a SQL Warehouse?
A SQL Warehouse is compute specialized for running SQL against the lakehouse. It runs the Photon engine and is tuned for concurrent SQL queries and dashboards. It comes in classic and serverless flavors.
-- Connected to a SQL Warehouse from the Databricks SQL editor
-- or from BI tools like Power BI, Tableau
SELECT
b.branch_name,
COUNT(*) AS txn_count,
SUM(t.amount) AS total_amount,
AVG(t.amount) AS avg_amount
FROM banking_prod.transactions t
JOIN banking_prod.branches b ON t.branch_code = b.branch_code
WHERE t.txn_date >= '2025-06-01'
GROUP BY b.branch_name
ORDER BY total_amount DESC;
Classic SQL Warehouse:
- Runs in YOUR cloud account (VPC)
- You manage sizing and scaling
- Startup: 2-5 minutes
Serverless SQL Warehouse:
- Runs in Databricks' managed pool
- Starts in seconds (pre-warmed)
- Auto-scales concurrency
- Startup: ~10 seconds
When do you use each cluster type?
The discipline is to match the compute lifecycle to the workload.
| Workload | Cluster Type | Why |
|---|---|---|
| Development, exploration | All-Purpose | Warm, shared, interactive |
| Nightly ETL, scheduled jobs | Job cluster | Isolated, ephemeral, no idle cost |
| BI dashboards, ad hoc SQL | SQL Warehouse | Photon engine, concurrent SQL |
| ML model training (scheduled) | Job cluster | Reproducible, isolated |
| Collaborative notebook session | All-Purpose | Multiple users, real-time |
How do you control costs?
Cost control requires matching compute type to workload and setting proper auto-termination and scaling policies.
# Cost control best practices
# 1. Auto-terminate All-Purpose clusters
# Set in cluster config: autotermination_minutes = 30
# 2. Use cluster pools for faster Job cluster startup
# Pre-allocate warm instances so job clusters start in <1 minute
pool_config = {
"instance_pool_name": "bank-etl-pool",
"node_type_id": "i3.xlarge",
"min_idle_instances": 2,
"max_capacity": 20,
"idle_instance_autotermination_minutes": 15
}
# 3. Use serverless SQL Warehouse for bursty BI
# Zero cost when no queries, instant scale-up when analysts arrive
# 4. Tag clusters for cost attribution
cluster_tags = {
"team": "data-platform",
"project": "transaction-pipeline",
"environment": "production"
}
What are cluster pools?
Cluster pools keep a set of warm, ready-to-use instances. When a Job cluster starts, it draws instances from the pool instead of provisioning new ones from the cloud provider, reducing startup from minutes to under a minute.
// Cluster pool configuration
{
"instance_pool_name": "bank-production-pool",
"node_type_id": "i3.xlarge",
"min_idle_instances": 4,
"max_capacity": 50,
"idle_instance_autotermination_minutes": 20
}
What about serverless vs classic for regulated banks?
Serverless warehouses minimize idle cost and startup latency but run outside your cloud account's network boundary, which some regulated environments restrict. Banks with strict data residency or network isolation requirements sometimes choose classic warehouses inside their own VPC.
Classic: Bank VPC ──► Classic Warehouse ──► Data in Bank S3
Serverless: Bank VPC ──► Databricks Managed ──► Data in Bank S3
(crosses network boundary)
What should a senior engineer know?
- Using an always-on All-Purpose cluster for a nightly production job wastes money and risks interference from other users.
- Using a Job cluster for interactive development frustrates users with repeated startup waits.
- Serverless SQL Warehouses are the best choice for bursty BI patterns where analysts arrive in the morning and leave in the evening.
- Cluster pools bridge the gap between ephemeral Job clusters (isolation) and fast startup (no wait).
- Auto-scaling policies on SQL Warehouses should match your concurrency SLA.
# Anti-pattern: running production ETL on a shared dev cluster
# This risks noisy-neighbor interference and has no isolation
# Correct: use a dedicated Job cluster for each production job
# and an All-Purpose cluster only for interactive development
Visual Overview
Databricks Compute Types: ┌─────────────────────────────────────────────┐ │ All-Purpose Cluster │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ User A │ │ User B │ │ User C │ │ │ │notebook │ │notebook │ │notebook │ │ │ └────┬────┘ └────┬────┘ └────┬────┘ │ │ └───────────┼───────────┘ │ │ ▼ │ │ Shared Spark cluster │ │ (long-lived, interactive) │ └─────────────────────────────────────────────┘ ┌─────────────────────────────────────────────┐ │ Job Cluster (ephemeral) │ │ Job triggers → Cluster CREATED │ │ ▼ │ │ Isolated Spark cluster │ │ ▼ │ │ Job completes → TERMINATED │ └─────────────────────────────────────────────┘ ┌─────────────────────────────────────────────┐ │ SQL Warehouse │ │ ┌──────┐ ┌──────┐ ┌──────┐ │ │ │ SQL │ │Tableau│ │PowerBI│ │ │ │editor│ │ │ │ │ │ │ └──┬───┘ └──┬───┘ └──┬───┘ │ │ └────────┼────────┘ │ │ ▼ │ │ Photon SQL Engine │ │ (serverless or classic) │ └─────────────────────────────────────────────┘
Key Takeaway
Databricks offers three main compute types for three different kinds of work. All-Purpose clusters are long-lived, shared, interactive clusters that you keep running so people can attach notebooks, ideal for development and exploration. Job clusters are ephemeral: created at the moment a scheduled job starts, running in isolation, and terminated the instant it finishes, so production pipelines do not interfere with each other and you pay only for the run duration. SQL Warehouses are compute endpoints specialized for SQL analytics and BI, running the Photon engine and tuned for concurrent queries.
The discipline is to match compute lifecycle to workload. Develop on All-Purpose clusters with auto-termination set. Run production jobs on ephemeral Job clusters. Serve dashboards on SQL Warehouses, preferring serverless for bursty BI because it starts instantly and scales automatically.
At the National Bank of India we initially ran everything on one always-on cluster, and the ETL collided with developers' queries while the cluster sat idle and expensive during quiet hours. We split workloads: developers on a shared All-Purpose cluster with 30-minute auto-termination, the nightly ETL on a Job cluster that spun up per run, and the morning dashboards on a serverless SQL Warehouse. That matching improved both reliability and cost.
The senior nuance is that serverless warehouses run outside your cloud account's network boundary, which some regulated banks restrict. Cluster pools give Job clusters near-instant startup without sacrificing isolation.
Remember one sentence: All-Purpose for development, Job clusters for production, SQL Warehouses for analytics, and match the compute lifecycle to the workload to control cost and isolation.
---