It seems like the AI crawlers learned how to solve the Anubis challenges. Anubis is a tool hosted on our infrastructure that requires browsers to do some heavy computation before accessing Codeberg again. It really saved us tons of nerves over the past months, because it saved us from manually maintaining blocklists to having a working detection for "real browsers" and "AI crawlers".

Hacker News is a community-driven platform for sharing and discussing technology news, startups, and programming-related topics. Through user submissions and comments, Hacker News offers insights into emerging technology trends, industry developments, and entrepreneurial ventures. Readers can participate in discussions, share their insights, and stay informed about the latest advancements in technology and innovation.

Hacker News

Codeberg experienced severe performance issues when AI crawlers evolved to bypass their Anubis protection system, which uses computational challenges to distinguish real browsers from bots. The crawlers, including those from Huawei networks, learned to solve these challenges and overwhelmed the platform. A configuration oversight allowed blocked IP ranges to access Anubis-protected routes, compounding the problem. After identifying and fixing the issue, Codeberg restored normal performance, but the incident highlights the ongoing arms race between platforms and increasingly sophisticated AI crawlers.

Codeberg: "It seems like the AI crawlers …"