In this article, we will look at how the Agoda engineering team built a single source of truth for its financial data and the challenges encountered.

ByteByteGo provides tutorials, articles, and resources for learning and mastering the Go programming language, covering topics such as syntax, concurrency, and best practices. Developers can learn about Go programming fundamentals, web development with Go, and building scalable applications using Go's powerful features and standard library.

ByteByteGo

Agoda faced data inconsistencies and reporting conflicts from multiple independent financial data pipelines owned by different teams. To solve this, they built FINUDP (Financial Unified Data Pipeline) on Apache Spark, creating a single source of truth for all financial metrics. Key practices include shadow testing, staging environments, multi-level alerting (email, Slack, GoFresh), data contracts with upstream teams, third-party quality validation via Quilliup, and ML-based anomaly detection. Major challenges included stakeholder alignment, a 5-hour runtime reduced to 30 minutes through optimization, and reconciling inconsistent metric definitions across legacy pipelines. The system achieved 95.6% uptime with a target of 99.5% data availability.

How Agoda Built a Single Source of Truth for Financial Data

Get first access to the Second Edition of Designing Data-Intensive Applications [O’REILLY] (Sponsored)

The Problems of Multiple Financial Data Pipelines

Unblocked: Context that saves you time and tokens (Sponsored)

The Solution: Financial Unified Data Pipeline

Technical Practices for Quality Assurance