Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This information can help site owners make decisions about which AI bots to allow or block, and enables users to understand how AI usage in aggregate impacts Internet traffic.

Cloudflare's platform is a leading provider of internet security and performance solutions, offering insights into web security, content delivery, and DNS management. Through documentation, blog posts, and webinars, Cloudflare provides insights into protecting websites and applications from cyber threats and improving performance. Developers and IT professionals can learn about CDN (Content Delivery Network), DDoS mitigation, and firewall configurations to secure and accelerate web traffic.

Cloudflare

Cloudflare Radar introduces a new metric showing crawl-to-referral ratios for AI platforms, revealing that AI bots crawl content far more frequently than they send traffic back to publishers. While traditional search engines maintained balanced ratios, AI platforms like Anthropic's Claude show ratios as high as 70,900:1, meaning they make nearly 71,000 page requests for every referral they generate. This imbalance threatens content publishers' revenue models, as AI systems consume and repackage content without driving meaningful traffic back to original sources. The data helps site owners make informed decisions about blocking AI crawlers and provides insights into how AI usage impacts internet traffic patterns.

The crawl before the fall… of referrals: understanding AI’s impact on content providers