Empowers leading publishers and AI companies to stop the scraping and use of original content without permission

Materialize

Bobby Iliev

DevOps

Community Picks is a section on daily.dev where our community members share the most interesting and valuable content they've discovered online. From insightful articles to handy tools, every post is a gem curated by our dedicated coomunity. To contribute to Community Picks, you need to have at least 250 reputation points, ensuring that only active and trusted members can share their finds.

Community Picks

Cloudflare has introduced a permission-based model that blocks AI crawlers from accessing website content without explicit consent by default. The new system requires AI companies to obtain permission before scraping content and allows website owners to control which crawlers can access their sites. This shift addresses the broken economic model where AI systems extract content without compensating creators or driving traffic back to original sources. Major publishers and media companies support this initiative, which includes a Pay Per Crawl beta program and enhanced bot authentication mechanisms.

Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model

<p>I am waiting for a price tag of this feature. I’ve been told its for enterprise customers with thousands usd monthly bill …</p>
