At my current company, one of Australia’s leading beverage companies, data is at the heart of nearly everything we do. In fact, about 80% of all the data we handle comes directly from SAP systems…

Data Engineer Things

A detailed walkthrough of building a scalable AWS-based data platform for an Australian beverage company handling diverse data sources including SAP systems, third-party vendors, and static files. The architecture uses AWS Step Functions to orchestrate three independent state machines for ingestion, processing, and loading data, with AWS Glue for ETL operations and Redshift for warehousing. Key design decisions include separating concerns across state machines, treating metadata capture as first-class, and leveraging visual orchestration for better troubleshooting.

How We Built the AWS Data & Analytics Platform (Part 1)

Processing: same state machine, new purpose