Pipeline cost optimization

In Databricks, the bill is the price of the cloud machine plus the DBUs (*Databricks Units*) consumed per hour, which vary with the compute type. Simplified: cost ≈ number of machines × hours on × price per hour. Every optimization touches one of these three.

Read less

• A filter that skips files: partitioning or clustering by the most filtered column, and the filter written in a way the optimizer can push down (no function wrapped around the column).
• Columns: select only what will be used; Parquet/Delta are columnar.
• Incremental instead of full: process only what arrived since the last load (Auto Loader, Change Data Feed, MERGE), not the whole table every night.
• Right-sized files: OPTIMIZE against small files.

Run on the right compute

• Jobs on job compute or serverless, never on someone's interactive cluster.
• Short auto termination on interactive clusters.
• Spot on the workers of jobs that tolerate restarts.
• Measured size, not guessed: doubling the workers only helps until the work stops being parallel.

Why "more machines" is not always more expensive, nor faster

If the work is highly parallel, twice the machines finish in half the time and cost the same. The part that does not parallelize (driver, startup, a skewed partition) does not shrink, and beyond it each extra machine is pure cost.

def estimate(workers, parallel_hours=8.0, fixed_hours=0.25, node_hour_price=1.0):
    time = fixed_hours + parallel_hours / workers
    cost = (workers + 1) * time * node_hour_price   # +1: the driver
    return round(time, 2), round(cost, 2)

for w in (2, 4, 8, 16, 32):
    print(w, estimate(w))

2 (4.25, 12.75)
4 (2.25, 11.25)
8 (1.25, 11.25)
16 (0.75, 12.75)
32 (0.5, 16.5)

It is a toy model, with a made-up price of 1 unit per machine-hour, and it shows the shape of the curve: time drops less and less, and cost, past a point, goes up. The simulator on the home page lets you play with these numbers.

• Tag clusters and jobs (team, project): without knowing who spends, there is no way to reduce.
• Databricks system tables (system.billing.usage) show consumption per job, with SQL.