CI/CD for data

CI/CD for data follows the same principle as software: every change goes through review and automated tests before it reaches production, and deployment is a repeatable process. The difference is that, besides the code, there is the data, which changes on its own.

The stages

• Lint and formatting: ruff or flake8 for Python, sqlfluff for SQL. Catches silly mistakes before spending cluster time.
• Unit tests: the transformation logic as a pure function, tested with small, known data.
• Deployment to dev/staging: the job goes up in an isolated environment.
• Integration and quality tests: runs on a realistic sample and checks the rules (see Data quality).
• Production: the same definition, with a different configuration, deployed by the pipeline (GitHub Actions, Azure DevOps, GitLab CI, Jenkins).

A testable transformation

The trick is to separate the rule from Spark: the function takes data and returns data.

def classify(amount):
    if amount is None:
        return "no amount"
    return "high" if amount >= 500 else "low"

def test_classify():
    assert classify(None) == "no amount"
    assert classify(500) == "high"
    assert classify(499.99) == "low"

test_classify()
print("tests ok")

tests ok

In PySpark, the same test with a small DataFrame runs on a local SparkSession inside pytest, without a cluster.

Databricks Asset Bundles

A *bundle* describes jobs, pipelines and settings in YAML (databricks.yml), versioned together with the code, with one target per environment. The CI pipeline validates and deploys:

bundle:
  name: medallion
targets:
  dev:
    mode: development
    default: true
  prod:
    mode: production

The CI sequence is usually databricks bundle validate, databricks bundle deploy -t dev, the tests, and databricks bundle deploy -t prod after approval.

• Production runs as a *service principal*, not as the user who did the merge.
• Secrets go in a vault (Azure Key Vault, secret scope), never in the notebook or in the YAML.