Best of workflow-orchestration — 2025

1. 1
   Article

   ByteByteGo·52w

   How Netflix Orchestrates Millions of Workflow Jobs with Maestro

   Netflix transitioned from using the Meson orchestrator to Maestro due to scalability issues with the growing volume of data and workflows. Maestro, built with a distributed microservices architecture, efficiently manages large-scale workflows with high reliability and low operational overhead. It supports dynamic workflows, defined via DSLs, a visual UI, or programmatic APIs, and leverages technologies such as CockroachDB and distributed queues. Features like event publishing, parameterized workflows, and an integrated signal service enable Maestro to handle extensive data processing and machine learning tasks at scale.

   107
2. 2
   Article

   Data Engineer Things·1y

   Its time to try Kestra

   Kestra is presented as an underrated yet powerful workflow orchestrator, boasting a user-friendly UI, YAML-based workflows, comprehensive documentation, and impressive scalability and performance. While it faces challenges such as being relatively new, having a smaller community, and some limitations in advanced features, Kestra’s simplicity and efficiency make it a promising tool for the future of data team workflow orchestration.

   105

   1
3. 3
   Article

   Medium·34w

   5 Agent Workflows You Need to Master (And Exactly How to Use Them)

   Five structured AI agent workflows are presented to replace ad-hoc prompting: prompt chaining breaks complex tasks into sequential steps, routing directs queries to appropriate models based on complexity, parallelization runs independent tasks simultaneously, orchestrator-workers use a planning model to coordinate specialized workers, and evaluator-optimizer creates feedback loops for quality improvement. Each workflow includes Python code examples and addresses specific use cases like code generation, content creation, and data analysis to achieve more consistent and production-ready results.

   81

   1
4. 4
   Article

   Towards Data Science·35w

   LangGraph 101: Let’s Build A Deep Research Agent

   A comprehensive tutorial on building AI research agents using LangGraph, Google's open-source framework. Covers core concepts including graph-based workflow modeling with nodes and edges, state management for agent memory, structured outputs for reliable LLM responses, tool calling for web searches, conditional routing for decision-making, and parallel processing for concurrent operations. Uses Google's Deep Research Agent implementation as a practical example, demonstrating how to create agents that can autonomously search the web, evaluate results, and generate comprehensive reports with citations.

   29
5. 5
   Article

   Data Engineer Things·1y

   Deep Dive into Airflow DAGs: Understanding the Core of Workflow Orchestration

   Explore the core concepts of Directed Acyclic Graphs (DAGs) in Apache Airflow, including their definition, key components, creating dynamic DAGs, and configuring them efficiently. Learn how to integrate Python scripts and use operators like PythonOperator and BashOperator to build scalable workflows.

   26

   1
6. 6
   Article

   Stack Overflow Blog·49w

   Mastering microservices with a former Uber and Netflix architect

   Jeu, a former architect at Uber and Netflix, now cofounder of Orkes, shares insights on mastering microservices. Orkes offers a developer-first enterprise workflow orchestration platform. The post highlights contributions by Stack Overflow user Alex Stiff.

   25
7. 7
   Article

   Netflix TechBlog·28w

   100X Faster: How We Supercharged Netflix Maestro’s Workflow Engine

   Netflix redesigned their Maestro workflow orchestrator engine, achieving 100x performance improvement by replacing the stateless worker model with a stateful actor-based architecture using Java virtual threads. The new design reduces overhead from seconds to milliseconds, maintains in-memory state for better locality, implements strong execution guarantees, and simplifies the architecture by removing dependencies on external distributed queues and multiple databases.

   17
8. 8
   Article

   Javarevisited·40w

   Top 8 Udemy Courses to Learn Apache Airflow in 2025

   A curated list of 8 Udemy courses for learning Apache Airflow in 2025, ranging from beginner to advanced levels. The courses cover workflow orchestration, DAG creation, cloud deployment, and production-level implementations. Recommendations include Marc Lamberti's hands-on introduction for beginners and advanced courses covering AWS, Docker, and Kubernetes integration for experienced users.

   16