Agoda consolidated multiple financial data pipelines into a single centralized system called FINUDP to address data inconsistencies, quality issues, and reliability concerns. The unified pipeline processes millions of daily financial data points using Apache Spark, achieving 95.6% uptime with a goal of 99.5%. Key improvements include hourly data updates, automated quality checks using tools like Quilliup and GoFresh, shadow testing, staging environments, and data contracts with upstream teams. The centralization reduced pipeline runtime from five hours to 30 minutes while establishing a single source of truth for financial metrics across the organization.
Table of contents
IntroductionThe Challenges of Multiple Financial Data PipelinesWhat Happens When Pipeline Inconsistencies Go UncheckedThe Solution: Centralized Financial Data Pipeline (FINUDP)Key challenges encountered while building FINUDPHow can we ensure data quality and uptime with a centralized data pipeline?Architectural Trade-offs in Centralizing Financial Data PipelinesConclusionSort: