Uber processes vast amounts of data daily—across multiple verticals—using technologies like Apache Hadoop™, Apache Hive™, and Apache Spark™. Each data team at Uber must operate within resource constraints while managing ever-growing data volumes. Our team, CDS (Compliance Data Store) serves as Uber’s central repository for regulatory reporting. We share data with regulators in accordance with applicable laws and requirements.. Moreover, managing this extensive data poses significant challenges, as we rely on HDFS™ (Hadoop Distributed File System), which imposes storage quotas like NSQ (Namespace Quota) for file and directory limits and a space quota for overall storage capacity.

The Uber Engineering Blog offers insights, technical deep dives, and updates on the engineering challenges and solutions behind Uber's platform and services. Covering topics such as distributed systems, data infrastructure, and mobile development, the blog provides resources for developers interested in large-scale systems and real-world engineering problems. Developers can learn about Uber's technology stack, engineering culture, and best practices for building scalable and reliable systems.

Uber Engineering

Uber's Compliance Data Store team developed a config-driven archival and retrieval framework to manage terabytes of regulatory data efficiently. The system automatically moves data from hot storage (HDFS) to cold storage (S3) based on configurable policies, while providing on-demand retrieval capabilities. The solution reduced manual intervention by 90% and successfully handles over 500 regulatory reports, addressing challenges like schema evolution, data consistency during backfills, and resource optimization. The framework uses MySQL for metadata management, Apache Airflow for orchestration, and includes a user-friendly interface for self-service data retrieval.

From Archival to Access: Config-Driven Data Pipelines