Cloudflare and ETH Zurich have identified that AI bot traffic—now exceeding 10 billion requests per week—is severely disrupting traditional CDN and database caching strategies. AI crawlers exhibit 70–100% unique URL access ratios, causing high cache miss rates and displacing human-oriented cached content. Unlike human users, AI bots don't leverage browser caching or session continuity, and their parallel, high-volume requests break assumptions underlying LRU eviction, prefetching, and cache speculation. The proposed solutions include separating human and AI traffic into distinct cache tiers, testing alternative eviction algorithms (LFU, FIFO), adopting ML-driven adaptive caching policies, and exploring pay-per-crawl models to control AI access while preserving cache efficiency.
Sort: