Compute¶

Serverless¶

3 types
- Standard: multi-user with shared resources, cost-effective, uses Lakeguard to ensure isolation
- Dedicated: assigned to single user or a group
- instance pools: pre-configured instances that reduce compute startup time
- Classic:
All purpose: ETL, ML, interactive (notebooks)
Jobs Computer: provisioned, scheduled, batch; used for running workflows and pipelines
Serverless: run workloads without provisioning a cluster, instant availability. Specialized for:
- SQL Warehouse: run SQL queries
- Notebooks: execute SQL and Python code in notebooks
- Jobs: running databricks jobs, pipelines, workflows
Vector search: for running vector search workloads
Instance Pools: pool of clusters, manage and scale clusters
- Databricks doesn't charge (DBU) for idle instances, but cloud-provider charges apply
- Each customer creates an instance pool
Classic SQL Warehouse: traditional SQL warehouse, requires cluster provisioning
Apps: interactive, Python or other apps

used for isolating workloads of multiple users when they share the same classic compute
uses Spark connect to decouple client applications to run on different JVM than the driver.
each client runs in a container environment
isolates UDF execution (by default executors do not isolate UDFs)

feature	Classic	Pro	Serverless
Predictive IO	-	Yes	Yes
Intelligent Workload Mgt	-	-	Yes
streaming tables	Yes	Yes	Yes
MV, QF, WFI ¹	-	Yes	Yes
Data Science/ML ²	-	Yes	Yes
AI/BI	Dashboards	Dashboards+Genie	Dashboards+Genie

serverless
- compute plane created by Databricks in customer's databricks account and in the same region as the Workspace
- runs within the network boundary of the Workspace