Jobs and Workflows: orchestration on Databricks

On Databricks, the native orchestrator is Jobs (for a long time called Workflows; today it shows up as *Lakeflow Jobs*). A job is a DAG of tasks: each task runs a notebook, a Python script, a wheel package, a SQL query, a declarative pipeline or another job, and only starts once the tasks it depends on have finished.

A medallion pipeline as a job

• bronze_ingestion (no dependency)
• silver_cleaning depends on bronze_ingestion
• gold_sales and gold_customers depend on silver_cleaning and run in parallel
• quality depends on both gold tasks; if it fails, the job fails and sends an alert

resources:
  jobs:
    daily_medallion:
      name: daily-medallion
      schedule:
        quartz_cron_expression: "0 30 5 * * ?"
        timezone_id: America/Sao_Paulo
      tasks:
        - task_key: bronze_ingestion
          notebook_task: {notebook_path: ../notebooks/bronze.py}
        - task_key: silver_cleaning
          depends_on: [{task_key: bronze_ingestion}]
          notebook_task: {notebook_path: ../notebooks/silver.py}
          max_retries: 2
        - task_key: gold_sales
          depends_on: [{task_key: silver_cleaning}]
          notebook_task: {notebook_path: ../notebooks/gold_sales.py}

(Databricks Asset Bundle format; see CI/CD for data.)

Quartz cron: six fields, with seconds

The job schedule uses Quartz syntax: seconds minutes hours day-of-month month day-of-week [year]. One of the two day fields takes ? ("don't care").

• 0 30 5 * * ?: every day at 05:30:00.
• 0 0 */2 ? * MON-FRI: every two hours, Monday to Friday.
• 0 0 6 1 * ?: on day 1 of every month, at 06:00.

The "cron" tool on this site translates an expression and lists the next runs.

What to configure besides the schedule

• Retries (max_retries) and the interval between them: a transient failure (network, lost node) doesn't need a person.
• Timeout per task: a stuck job costs an idle cluster.
• Alerts for failure and for abnormal duration.
• Parameters (reference date) instead of a hard-coded "today" in the code: reprocessing a day becomes running the job with a different parameter.
• File-arrival trigger or table-update trigger, when waiting for the clock is wasteful.