We previously published a blog about how we scaled adoption of MIT Kerberos™ at Uber. We built an automation system called KDP (Keytab Distribution Pipeline) that generates and distributes Kerberos keytabs (credentials) to systems that must authenticate with Kerberos. With the help of this system, we drove adoption of Kerberos authentication for several critical use cases. Some of the key use cases include: fetching search indexes from Apache HDFS™ that powers Uber Eats search, enabling Apache Flink® security for several hundreds of streaming analytics applications, authentication for all 250 Apache Zookeeper™ clusters used by data infrastructure, and batch analytics infrastructure authentication comprising of 20+ systems.

The Uber Engineering Blog offers insights, technical deep dives, and updates on the engineering challenges and solutions behind Uber's platform and services. Covering topics such as distributed systems, data infrastructure, and mobile development, the blog provides resources for developers interested in large-scale systems and real-world engineering problems. Developers can learn about Uber's technology stack, engineering culture, and best practices for building scalable and reliable systems.

Uber Engineering

Uber automated the rotation of over 100,000 Kerberos keytabs across their infrastructure by building a custom system that integrates with their Secret Management Platform. The solution addresses key challenges including scale complexity and service disruption risks through failure-domain-aware automation, rate limiting, and careful timing coordination. The system minimizes authentication failures during rotation by ensuring sufficient TGT validity time and maintaining both old and new key versions on servers during transitions. Safety measures include blast radius minimization, cluster-based allowlists for gradual rollout, and automatic deletion capabilities. Uber is now exploring PKINIT as the next evolution to replace keytabs with certificate-based authentication.

Automating Kerberos Keytab Rotation at Uber