Uber has been running an open-source Apache Cassandra® database as a service that powers a variety of mission-critical OLTP workloads for more than six years now at Uber scale, with millions of queries per second and petabytes of data. As Uber operates data centers in multiple zones across multiple regions, a Cassandra cluster at Uber typically has its nodes spread across multiple zones and regions. With high availability being essential for Uber’s business, we’d like to have Cassandra’s availability unaffected in the scenario of a single-zone going down. This blog shows how we ensured the single zone failure tolerance for Cassandra, and particularly how we converted the large Cassandra fleet in real-time with zero downtime from non-zone-failure tolerant to single-zone-failure tolerant.

The Uber Engineering Blog offers insights, technical deep dives, and updates on the engineering challenges and solutions behind Uber's platform and services. Covering topics such as distributed systems, data infrastructure, and mobile development, the blog provides resources for developers interested in large-scale systems and real-world engineering problems. Developers can learn about Uber's technology stack, engineering culture, and best practices for building scalable and reliable systems.

Uber Engineering

Uber has successfully enhanced its Apache Cassandra database deployment to ensure it can tolerate single-zone failures, maintaining high availability despite outages in specific zones. They achieved this by transitioning from a single-rack to a multi-rack setup, rebuiling clusters with real-time data synchronization, and enhancing Cassandra clients for seamless traffic switching. This complex process involved zero downtime and required no changes from Uber's stakeholders.

How Uber ensures Apache Cassandra®’s tolerance for single-zone failure