<h1>Cloudflare Denounces Perplexity AI’s Crawling Tactics</h1>
Cloudflare has accused Perplexity AI of employing stealth tactics to bypass website restrictions, drawing significant criticism regarding internet crawling norms established over three decades ago. The AI search engine reportedly uses methods such as changing user agents to mimic browsers like Chrome, rotating IP addresses across different networks, and ignoring robots.txt files and firewall blocks. These techniques allow Perplexity to scrape content from tens of thousands of domains, resulting in millions of requests daily.
Perplexity’s actions starkly contrast with other AI entities, such as OpenAI, which adhere to website directives. When their declared crawlers face blocks, Perplexity allegedly shifts to undeclared crawlers, generating between three to six million requests per day. Cloudflare has responded by delisting Perplexity as a verified bot and has implemented detection rules to curb this stealth activity. They have also equipped their customers with tools to manage AI crawler access.
Perplexity’s deceptive tactics, including disguising its crawlers as Google Chrome browsers and utilizing IP rotation, have raised concerns. This behavior not only circumvents Web Application Firewall rules but also exemplifies controversial scraping practices. While Perplexity denies these allegations, labeling Cloudflare’s findings a sales pitch, Cloudflare maintains its stance, highlighting a broader issue within the industry.
The increasing divergence between website preferences and AI crawler activities is becoming significant. Bot violations of robots.txt files have surged, impacting the relationship between website owners and AI companies. This parasitic interaction, where AI entities extract content without offering compensation, is prompting companies like Cloudflare and TollBit to introduce paywall solutions.
Cloudflare’s confrontation with Perplexity serves as a cautionary tale for the evolving dynamics of AI-driven web interactions, emphasizing the need for respect towards established online protocols.

Collections

Cloudflare has accused Perplexity AI of using deceptive crawling tactics to bypass website restrictions, including changing user agents to mimic browsers, rotating IP addresses, and ignoring robots.txt files. This generates millions of daily requests across thousands of domains, contrasting with other AI companies like OpenAI that respect website directives. Cloudflare has delisted Perplexity as a verified bot and implemented detection rules, while also providing tools for customers to manage AI crawler access. The controversy highlights growing tensions between AI companies extracting content and website owners seeking compensation or control over their data.

Cloudflare and Perplexity AI: A Clash Over Stealth Web Crawling Practices