Orchestrating Success

Vinted solved the challenge of coordinating data pipelines across decentralized teams by building a DAG generator that automatically creates task-per-model Airflow pipelines from dbt manifests. They developed an Asset Registry to manage cross-domain dependencies using ExternalTaskSensor, enabling fine-grained task-level coordination without manual wiring. The system automatically handles late completions and timed-out sensors through completion callbacks, while providing CI/CD visibility into downstream dependencies. This approach standardizes pipeline creation, eliminates manual DAG authoring, and allows platform-wide changes like the Airflow 3 upgrade to be rolled out transparently.