Shopify deprecated its admin REST API in favor of GraphQL, requiring data pipeline migrations. This walkthrough covers rebuilding a Shopify-to-Snowflake pipeline using Openflow (Apache NiFi) and Shopify's GraphQL Bulk Operations API. Instead of paginating through thousands of REST requests, the new approach submits a bulk export job, polls for completion, downloads the resulting JSONL file, and loads it into a Snowflake internal stage. Key design patterns include using flow files as state machines to track job progress, Parameter Contexts for environment-agnostic configuration, and Controller Services for reusable Snowflake connections. The post also covers Snowflake network rules and external access integrations for both Shopify and Google Cloud Storage, and notes limitations like asynchronous processing, rate limits, and fixed JSONL output format.
Table of contents
When “Simple” Stops ScalingWhat we are building:A Small But Important Mindset ShiftWhat The Architecture Looks LikeFlow Files Are State MachinesParameter Contexts Keep Things CleanGet DW Conradie’s stories in your inboxController Services Reduce ComplexityHow the Pipeline Flows:Limitations and CaveatsStep 1: Configure OpenflowStep 2: Build the PipelineSort: