Workflow orchestration is one of the most important skills in data engineering. And also one of the hardest to get right in production.

In this free full course video, you’ll learn how to build scalable, reliable workflows with Kestra on Google Cloud Platform (GCP). We start locally with Docker and move step by step toward real production-style pipelines using Cloud Storage (GCS), BigQuery, and event-driven triggers.

This is a hands-on, end-to-end walkthrough. You’ll learn how Kestra workflows (flows) are defined in YAML, how tasks exchange inputs and outputs, and how to design pipelines that stay maintainable as they grow.

Throughout the course, you’ll:
▶️ Run real Python and Java code inside Kestra workflows
▶️ Synchronize workflows and code from GitHub using Namespace & Tenant Sync
▶️ Build reusable pipelines with subflows
▶️ Load and transform data with BigQuery + GCS
▶️ Process multiple files dynamically using loops
▶️ Scale execution with parallel tasks
▶️ Implement error handling, fallbacks, and retries
▶️ Build event-driven pipelines triggered by file uploads (Pub/Sub)
▶️ Monitor, debug, replay workflows, and build dashboards with execution metrics

If you found this useful, leave a like or a comment! It helps the channel grow and tells me what to cover next.

YouTube

A comprehensive course on building scalable data pipelines using Kestra on Google Cloud Platform. Covers setting up Kestra locally via Docker, connecting to GCP services (GCS and BigQuery), running Python and Java code within workflows, GitHub synchronization for namespace and tenant sync, subflows for reusability, looping over multiple files with concurrency, parallel execution patterns, and event-driven pipelines. Emphasizes the key principle that Kestra should orchestrate external systems rather than perform heavy data processing itself.

FREE COURSE: How to Build Scalable Data Pipelines with Kestra on GCP