Evil Martians migrated Wallarm's core event pipeline from NATS to Kafka in two months with zero downtime. Learn how we also handle event deduplication and reconstruct business flows for better understanding of the application.

Evil Martians offers insights into software development, web design, and digital product strategy, providing articles, case studies, and open-source contributions to help businesses innovate and deliver exceptional user experiences. By exploring Evil Martians' curated content, developers can learn about Ruby on Rails development, frontend frameworks, and modern web technologies for building scalable and maintainable applications. Whether you're launching a new product, redesigning a website, or optimizing user interfaces, Evil Martians offers resources to bring your ideas to life and achieve digital success.

Martian Chronicles

Evil Martians migrated Wallarm's core event pipeline from NATS to managed Kafka in two months with zero downtime, handling peak loads of 30k RPS. The work also included fixing event deduplication in ClickHouse using materialized views for exactly-once semantics, building an internal URL parameterization library for developer experience, and creating a Python-based pipeline that reconstructs business flows from raw API traffic using LLM-assisted naming.

Migrating an event pipeline from NATS to Kafka with zero downtime—Martian Chronicles, Evil Martians’ team blog

Part I: cleaning up ClickHouse and removing duplicates

Part 2: migrating NATS to Kafka in two months, building internal libraries, and analyzing API traffic