By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling. GPTBot and ClaudeBot surged, Amazonbot and Bytespider collapsed, and crawl-to-refer ratios show AI consumes far more than it sends back.

Cloudflare's platform is a leading provider of internet security and performance solutions, offering insights into web security, content delivery, and DNS management. Through documentation, blog posts, and webinars, Cloudflare provides insights into protecting websites and applications from cyber threats and improving performance. Developers and IT professionals can learn about CDN (Content Delivery Network), DDoS mitigation, and firewall configurations to secure and accelerate web traffic.

Cloudflare

Cloudflare's data reveals a significant shift in web crawling patterns during 2025, with AI training bots now accounting for nearly 80% of AI crawling activity while referral traffic from search engines to publishers has declined. Major AI crawlers like GPTBot and ClaudeBot have surged in activity, while others like Bytespider have collapsed. The analysis shows extreme imbalances in crawl-to-referral ratios, with some AI platforms crawling tens of thousands of pages for every visitor they send back to content creators, highlighting concerns about AI systems consuming web content without providing proportional value in return.

The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals

Google referrals fall as AI Overviews expand

AI and search crawling: spring surge (+24%), summer slowdown

AI-only crawlers: OpenAI rises, ByteDance falls

Crawl-to-refer ratios shifts: tens of thousands of bot crawls per human click