Delta Lake: ACID transactions on top of files

A data lake of loose Parquet files has no transactions: a job that fails halfway leaves half the files written, and anyone reading at that moment sees half-finished data. Delta Lake solves this with a transaction log next to the data.

What is inside a Delta table

• data files in Parquet;
• the _delta_log/ folder, with one JSON file per commit (00000000000000000000.json, …01.json, …) and, from time to time, a *checkpoint* in Parquet that summarizes the state so a read does not have to replay every JSON file.

Each commit lists the actions: files added (add), files removed (remove), schema change (metaData). The current version of the table is the result of applying the log in order.

ACID in practice

• Atomicity: a commit is the write of one log file. Either it exists (the write took effect in full) or it does not (nothing changed for readers).
• Consistency: the schema lives in the log, and a write with a column of the wrong type is rejected (*schema enforcement*), unless you ask for evolution (mergeSchema).
• Isolation: a reader uses a fixed version of the log, so it never sees a half-done write. Concurrent writers use optimistic concurrency control: if two of them touch the same files, one commit fails with a conflict.
• Durability: the data and the log live in object storage (ADLS, S3, GCS).

(orders.write.format("delta")
    .mode("append")
    .saveAsTable("silver.orders"))

# schema with a new column: rejected, unless you ask for it
(orders_with_channel.write.format("delta")
    .mode("append")
    .option("mergeSchema", "true")
    .saveAsTable("silver.orders"))

Removing does not delete (yet)

DELETE and UPDATE do not rewrite the whole file in place: they mark old files as removed in the log and write new ones. The old files stay on disk until VACUUM. That is what makes it possible to go back in time (see Delta Lake time travel and history), and it is why the folder keeps growing if nobody runs VACUUM.

• On Databricks, Delta is the default table format.
• *Deletion vectors* (a newer feature) mark deleted rows without rewriting the file, making DELETE/UPDATE/MERGE cheaper.