Redirects for AI Training enforces canonical content

Cloudflare has launched 'Redirects for AI Training', a feature that automatically converts existing canonical tags into HTTP 301 redirects for verified AI training crawlers (GPTBot, ClaudeBot, Bytespider, etc.). The problem: advisory signals like noindex meta tags, deprecation banners, and canonical tags are routinely ignored by AI training crawlers, causing them to ingest outdated content. On developers.cloudflare.com, deprecated pages were crawled 46,000+ times by OpenAI alone in a single month, leading AI assistants to return outdated CLI syntax. The feature works by reading response HTML for non-self-referencing canonical tags and issuing a 301 before serving the response — with no origin changes required. It's available on all paid Cloudflare plans via a single toggle in AI Crawl Control. Human traffic and AI agents are unaffected. Cloudflare Radar's AI Insights page also now includes response status code analysis for AI crawler traffic, showing how the broader web responds to these crawlers.