A detailed walkthrough of building a scalable AWS-based data platform for an Australian beverage company handling diverse data sources including SAP systems, third-party vendors, and static files. The architecture uses AWS Step Functions to orchestrate three independent state machines for ingestion, processing, and loading data, with AWS Glue for ETL operations and Redshift for warehousing. Key design decisions include separating concerns across state machines, treating metadata capture as first-class, and leveraging visual orchestration for better troubleshooting.
Sort: