Once upon a time, during a normal Sunday, our team ran into an unexpected challenge: an Elasticsearch cluster that suddenly became sluggish and unresponsive due to a self-inflicted Denial of...

Zalando Tech Blog is a platform where Zalando's engineering team shares insights, experiences, and best practices on technology and innovation. Covering topics such as software architecture, data science, and machine learning, Zalando's Tech Blog offers resources for developers and tech enthusiasts. Developers can learn about Zalando's engineering culture, projects, and technology stack through their blog posts and articles.

Zalando

Zalando's Search & Browse team experienced a self-inflicted DoS attack when an internal application sent resource-intensive faceting queries on high-cardinality fields to their Elasticsearch cluster. The incident caused search slowdowns and empty results for customers. The team mitigated by splitting markets across clusters, implementing load shedding, and eventually traced the issue to a maintenance workload bug generating 50x normal query volume. Key lessons included improving per-client monitoring with X-Opaque-Id headers, implementing query-level rate limiting, adding aggregation size controls, and recognizing that performance issues can stem from unexpected sources rather than common causes.

The Day Our Own Queries DoS’ed Us: Inside Zalando Search

Additional Load Shedding: Making the Cluster Breathe Again

New Investigation and Finally, Root Cause

Some theory on Elasticsearch DoS via Faceting Queries on High Cardinality Fields

<p>I heard Zalando has the most toxic company culture in all of Europe. Yikes.</p>
