P99Conf's resource offers insights, tutorials, and resources for software developers and engineering leaders. Readers can learn about software engineering practices, system reliability, and distributed systems. With articles, talks, and case studies, P99Conf provides  guidance and expertise for building and managing high-performance systems.

P99 Conf

A detailed incident report about how a misconfigured Vertical Pod Autoscaler (VPA) caused widespread pod evictions in a Kubernetes cluster. The issue stemmed from the VPA Admission Controller being throttled by low API rate limits, preventing it from properly annotating pods and causing the Updater to continuously evict them. The problem was resolved by increasing the kube-api-qps and kube-api-burst parameters, highlighting the importance of understanding component internals and monitoring webhook performance.

When VerticalPodAutoscaler Goes Rogue: How an Autoscaler Took Down Our Cluster

The Incident: When VPA Started Evicting Everything