Skip to content

Compute

Serverless

Classic

  • 3 types
    • Standard: multi-user with shared resources, cost-effective, uses Lakeguard to ensure isolation
    • Dedicated: assigned to single user or a group
    • instance pools: pre-configured instances that reduce compute startup time
    • Classic:
  • All purpose: ETL, ML, interactive (notebooks)
  • Jobs Computer: provisioned, scheduled, batch; used for running workflows and pipelines
  • Serverless: run workloads without provisioning a cluster, instant availability. Specialized for:
    • SQL Warehouse: run SQL queries
    • Notebooks: execute SQL and Python code in notebooks
    • Jobs: running databricks jobs, pipelines, workflows
  • Vector search: for running vector search workloads
  • Instance Pools: pool of clusters, manage and scale clusters
    • Databricks doesn't charge (DBU) for idle instances, but cloud-provider charges apply
    • Each customer creates an instance pool
  • Classic SQL Warehouse: traditional SQL warehouse, requires cluster provisioning
  • Apps: interactive, Python or other apps

Lakeguard

  • used for isolating workloads of multiple users when they share the same classic compute
  • uses Spark connect to decouple client applications to run on different JVM than the driver.
  • each client runs in a container environment
  • isolates UDF execution (by default executors do not isolate UDFs)

SQL Warehouses

feature Classic Pro Serverless
Predictive IO - Yes Yes
Intelligent Workload Mgt - - Yes
streaming tables Yes Yes Yes
MV, QF, WFI (1) - Yes Yes
Data Science/ML (2) - Yes Yes
AI/BI Dashboards Dashboards+Genie Dashboards+Genie
  1. Materialized Views, Query Federation and Workflow Integration

  2. Data Science/ML: Python UDFs, Notebooks, GeoSpatial

  3. serverless

    • compute plane created by Databricks in customer's databricks account and in the same region as the Workspace
    • runs within the network boundary of the Workspace