4 YAML Files Instead of PySpark: How We Let Analysts Build Data Pipelines Without Engineers

A data engineering team at Mindbox replaced PySpark pipelines with a declarative stack using dlt, dbt, Trino, and Airflow+Cosmos, enabling analysts and product managers to build data pipelines using only YAML and SQL. The approach reduces delivery time from weeks to one day. The post walks through four YAML/config files: a dlt.yaml for data ingestion, dbt SQL models with sources.yaml and dbt_project.yaml for transformations, and a dag.yaml for Airflow orchestration. Limitations are also covered, including dlt's experimental Delta upsert support, Trino's fault tolerance constraints at terabyte scale, and the difficulty of custom UDFs outside of SQL.