At Airbnb, Apache Flink was introduced in 2018 as a supplementary solution for stream processing. It ran alongside Apache Spark™ Streaming for several years before transitioning to become the primary…

Airbnb's platform is  dedicated to providing insights and resources for developers and technology enthusiasts, focusing on engineering challenges, innovation projects, and best practices in software development. Through articles, tech talks, and open-source contributions, Airbnb offers insights into building scalable and reliable technology solutions. Developers can learn about Airbnb's engineering culture, technology stack, and contributions to the broader tech community.

Airbnb

Airbnb transitioned its stream processing architecture from Apache Hadoop Yarn to Kubernetes. This migration improved developer velocity, job availability, and infrastructure costs. The current setup integrates Flink directly with Kubernetes, offering a better developer experience, secure secrets management, isolated environments, enhanced monitoring, and simpler service discovery. The future focus will involve improving job availability, enabling autoscaling, and utilizing the Flink Kubernetes Operator for streamlined operations.

Apache Flink® on Kubernetes