Snowflake Task DAGs by default run all child nodes on the same schedule inherited from the root task. This tutorial presents a workaround for assigning different execution frequencies to individual child tasks within a single DAG. The solution uses a config table to persist a counter state and a stored procedure that increments and checks a modulo-based counter. Proxy 'counter' tasks are inserted before child tasks, calling the stored procedure and passing a boolean signal downstream via SYSTEM$SET_RETURN_VALUE. Child tasks then use a WHEN clause with SYSTEM$GET_PREDECESSOR_RETURN_VALUE to conditionally skip execution. The approach is demonstrated with an hourly root DAG where some reports run every 4 hours and others once daily, with full test-drive examples showing SKIPPED task history entries when conditions aren't met.

11m read timeFrom medium.com
Post cover image
Table of contents
Take Full Control of Your Pipeline Refresh FrequencyPart 0: Prep WorkPart 1: Create a Task DAGPart 2: Config Table and Stored ProcedurePart 3: Modified Task DAGGet Sabrina Liu’s stories in your inboxPart 4: Test Drive

Sort: