Cloudflare to block AI bots from scraping content by default

Cloudflare, a major content delivery network (CDN), has announced that it will block AI crawlers from accessing content without the consent or compensation of website owners.

This new policy will take effect for all new web domains signing up for its services starting July 2, 2025.

Website owners can now block AI bots or charge them using a “pay per crawl” model, expanding on Cloudflare’s 2024 one-click crawler-blocking tool.

Cloudflare estimates it handles 16% of global internet traffic, making this move significant for AI developers who rely on web data for training models.

CEO Matthew Prince said that the decision aims to empower creators while fostering innovation in AI.

AI crawlers from firms like OpenAI and Google often extract data without compensation, which Cloudflare says harms publishers’ ad revenues.

OpenAI, backed by Microsoft, criticized the policy, saying it adds a middleman and that its bots already follow standard web-scraping rules.

🔗 Source: CNBC

🧠 Food for thought

1️⃣ The economics of web content are fundamentally changing

The traditional internet economy operated on a simple exchange: search engines would crawl content, index it, and then direct users back to the original sites, generating traffic and ad revenue for creators.

This model is now being disrupted as AI crawlers extract content without providing return traffic, effectively breaking the economic feedback loop that sustained content creation 1.

Cloudflare’s move represents a significant intervention in this shifting landscape, with the company handling approximately 16% of global internet traffic according to their own 2023 report.

The introduction of “pay-per-crawl” marks the potential emergence of a new business model where AI companies must compensate publishers directly for training data, rather than offering indirect benefits through traffic 2.

This shift parallels other industries where raw materials have transitioned from freely available resources to commodities with pricing mechanisms as their economic value became more apparent.

The change addresses a growing concern among publishers that their content investments are subsidizing AI companies’ development without fair compensation, a sentiment that has driven recent high-profile lawsuits between news organizations and AI firms.