Kafka sits at the heart of how we move data between systems at trivago. Many teams publish changes to Kafka, and downstream services consume those changes to keep user-facing features up to date...

Trivago's platform is a central hub for technology enthusiasts and travel industry professionals, offering insights into travel technology, data analytics, and online booking platforms. Through articles, case studies, and tech talks, Trivago offers insights into building scalable and personalized travel search engines. Developers can learn about data science techniques, recommendation algorithms, and real-time data processing to improve user experience in the travel industry.

Tech at trivago

Trivago runs 50+ Kafka sink services across three regions (US, EU, ASIA) that materialize CDC events into service-local databases. Most sinks were idle the majority of the day yet consumed ~1 CPU core and 1 GB RAM each, wasting significant cluster capacity. CPU/memory-based autoscaling proved ineffective because sink workloads are I/O-bound and resource usage stays flat even when Kafka consumer lag grows. The solution was KEDA (Kubernetes Event-Driven Autoscaling) with its Kafka scaler, using consumer group lag as the scaling signal to scale deployments all the way down to zero replicas when idle. Key configuration parameters covered include `lagThreshold`, `activationLagThreshold`, `cooldownPeriod`, and `fallback`. An edge case around inactive consumer group cleanup for very low-traffic topics was solved by adding a Cron scaler to periodically wake sinks up. The rollout was gradual: one non-critical sink in one region first, then expanding after validation. Results: average daily consumption dropped from ~50 replica-hours per region to ~1–2 replica-hours.

From Always-On to On-Demand: Scaling Kafka Sinks with KED...

The Problem: Idle Sinks Burning Resources

Why Traditional Autoscaling Wasn’t Enough

Edge case: consumer group cleanup for very low-traffic topics