Uber’s HiveSync is a sharded, cross-region batch replication system keeping Hive/HDFS data consistent across multiple regions. Handling 5M daily Hive events and 8PB of data replication, it uses event-

InfoQ is a leading online platform for software developers, architects, and technical leaders, providing news, articles, presentations, and interviews on a wide range of topics, including agile practices, DevOps, microservices, and emerging technologies. With a focus on quality content and expert insights, InfoQ helps professionals stay informed about the latest trends, best practices, and industry developments. Developers can learn from real-world experiences, gain  knowledge, and connect with peers in the global software community through InfoQ's diverse and engaging content.

InfoQ

Uber built HiveSync, a sharded batch replication system that synchronizes 350PB of Hive/HDFS data across multiple regions. The system processes 5 million daily Hive events and replicates 8PB of data, using event-driven architecture, DAG-based orchestration, and hybrid RPC/DistCp strategies. HiveSync separates control and data planes, employs finite-state machines for job management, and includes Data Reparo for reconciliation. It maintains 99.99% cross-region accuracy with a 20-minute 99th percentile replication lag, enabling disaster recovery while eliminating idle secondary region costs.

350PB, Millions of Events, One System: Inside Uber’s Cross-Region Data Lake and Disaster Recovery