Best of ByteByteGo — October 2025

1. 1
   Article

   ByteByteGo·28w

   How Flipkart Built a Highly Available MySQL Cluster for 150+ Million Users

   Flipkart built Altair, an internally managed MySQL service that maintains high availability for 150+ million daily users through automated failover and primary-replica architecture. The system uses a three-layered monitoring approach (agent, monitor, orchestrator) to detect failures, prevent false positives, and execute failovers with minimal data loss. Altair prioritizes write availability over strong consistency using asynchronous replication, implements DNS-based service discovery for seamless failovers, and includes multiple safeguards against split-brain scenarios. The design balances operational simplicity with reliability, achieving near five-nines availability while managing thousands of database clusters across Flipkart's microservices infrastructure.

   120
2. 2
   Article

   ByteByteGo·29w

   How Airbnb Runs Distributed Databases on Kubernetes at Scale

   Airbnb deployed distributed SQL databases across multiple Kubernetes clusters, each mapped to a different AWS Availability Zone, to achieve high availability and fault tolerance. They built custom Kubernetes operators to safely manage stateful workloads, coordinate node replacements, and maintain quorum during failures. Using AWS EBS for persistent storage, PVCs for volume management, and techniques like replica reads and stale reads, they mitigated latency issues while maintaining consistency. Their largest production cluster handles 3 million queries per second across 150 nodes with 300TB of data, achieving 99.95% availability through careful sequencing of upgrades, canary deployments, and overprovisioning for resilience.

   67

   1
3. 3
   Article

   ByteByteGo·28w

   How OpenAI Uses Kubernetes And Apache Kafka for GenAI

   OpenAI built a stream processing platform using Apache Flink (PyFlink) on Kubernetes to handle real-time data for AI model training and experimentation. The architecture addresses three key challenges: providing Python-first APIs for ML practitioners, handling cloud capacity constraints, and managing multi-primary Kafka clusters. The system features a control plane for multi-cluster failover, per-namespace isolation in Kubernetes, watchdog services for Kafka topology monitoring, and decoupled state management using RocksDB with highly available blob storage. Custom Kafka connectors enable reading from multiple primary clusters simultaneously while maintaining resilience during outages.

   56