Compute: clusters, job clusters and serverless

All Spark code on Databricks runs on a *compute*: a driver and zero or more workers (executors). The type of compute decides how much you pay and for how long.

The types

• All-purpose (interactive): for notebooks and exploration. It stays on while someone is using it; configure auto termination so it shuts itself down after a few idle minutes.
• Job compute: created when the job starts and destroyed when it ends. It is the default path for production, with a lower price per DBU than interactive.
• Serverless: Databricks manages the machines; startup is fast and there is no cluster to size.
• SQL warehouse: compute for SQL queries and BI (Power BI connects here).

Sizing

• Autoscaling: minimum and maximum workers; the cluster grows with the task queue and shrinks when idle.
• Node type: memory for heavy shuffle and cache, CPU for simple transformation, local disk for large shuffle.
• Spot (spare-capacity instances, cheaper): good for workers, which can be replaced; the driver stays on a regular machine.
• Photon: a vectorized C++ engine that speeds up SQL and DataFrame; it costs more DBU per hour and usually pays the difference back in time.

Signs of a badly sized cluster

• Low CPU all the time with many workers: the cluster is too big, or the work is a single task (skew, coalesce(1), window without a partition).
• *Spill* to disk and out-of-memory errors: too little memory per core, or partitions that are too large.
• A short job on a cluster that takes minutes to start: startup time becomes a big part of the bill; pools or serverless fix it.

• A *compute policy* limits what each team can create: node type, maximum workers, mandatory auto termination. It is cost governance before the spending.