Cloudflare has accused AI search engine Perplexity of bypassing site restrictions meant to block its bots.

To test this, Cloudflare created a website that blocked Perplexity via robots.txt, yet found it was accessed using a generic browser user agent.

Perplexity denied that the activity came from its bots and said a third-party service may have been responsible.

The incident has sparked debate over whether AI agents accessing websites at a user’s request should be treated like automated bots or human visitors.

Cloudflare argued that other firms like OpenAI respect such restrictions, contrasting Perplexity’s behavior amid growing concerns over AI web scraping.

🔗 Source: TechCrunch

🧠 Food for thought

1️⃣ AI crawlers are fundamentally changing the economics of web traffic

The Perplexity controversy represents a broader shift from mutually beneficial web crawling to extractive data harvesting that doesn’t reward content creators.

Traditional search engines like Google created a symbiotic relationship. They crawled websites but sent users back to the original sources, generating revenue through clicks and ad views.

AI search engines operate differently, with data showing a significant drop in referral traffic to news sites and blogs compared to traditional search engines 1.

This means websites bear the infrastructure costs of being crawled, with some reporting 30TB of bandwidth consumed in a single month from AI crawlers 2, while receiving virtually no traffic in return.

The scale is notable: OpenAI and Perplexity scraped websites approximately 2 million times on average in Q4 2024, with AI crawlers now accounting for 28% of total web crawling traffic 13.

2️⃣ Website owners face an impossible choice between visibility and exploitation

The current AI crawling landscape forces website owners into a difficult scenario that threatens the open web’s future.

Blocking AI crawlers means missing out on visibility in what’s becoming a dominant information discovery channel, but allowing them means subsidizing competitors who profit from your content without compensation.