Data engineering is bogged down by orchestration and ops. Spark Declarative Pipelines (SDP) makes end-to-end pipelines declarative in Apache Spark.

databricks

Spark Declarative Pipelines (SDP), introduced in Apache Spark 4.1, extends declarative programming from individual queries to entire data pipelines. Instead of manually managing incremental processing, dependency ordering, data quality checks, backfills, and orchestration via external tools like Airflow, engineers declare what datasets should exist and SDP handles execution end-to-end. A side-by-side comparison of a weekly sales pipeline in PySpark, dbt, and SDP shows the declarative approach reducing boilerplate significantly—20 lines vs. much more verbose imperative code—while automatically inferring dependencies, handling incremental updates, validating pipelines upfront with dry-run, and quarantining bad records via inline expectations. The roadmap includes continuous execution, CDC support, and more efficient incremental processing developed openly with the Spark community.

Spark Declarative Pipelines: Why Data Engineering Needs to Become End-to-End Declarative