Bots, bills, and data scraping: Inside Cloudflare’s AI plan

This article summarizes an episode of Alex Kantrowitz’s video series featuring Matthew Prince, CEO of Cloudflare.

Image credit: Shutterstock

The old internet deal, where websites trade data for search traffic, is dead. It has been destroyed by AI systems that consume information without sending human visitors back.

Matthew Prince, co-founder and CEO of Cloudflare, argues that website owners must adapt to survive. They must handle the rising costs of automated servers, create financial models to make tech giants like Google pay for content, and turn their brands into digital credentials to stay visible in an AI-driven market.

AI traffic triggers an infrastructure crisis

As website owners adapt to this AI-driven reality, their first challenge is managing server expenses. The immediate threat to web publishers is the physical cost of hosting automated systems that never make a purchase, leaving independent sites struggling to survive server bills.

This imbalance requires technology providers to implement measures to prevent bankruptcy across the internet:

• Volume explosion: Automated traffic online is expected to be a thousand times greater than human traffic within five years, according to Prince.
• Hardware deficit: Current infrastructure cannot support this scale, as running a single AI agent in a traditional container would require more than 40 times the current CPU production.
• Usage limitations: Web hosts must deploy access controls to shield basic sites from corporate rates.

Monetizing data breaks Google’s special status

Once infrastructure costs are managed, publishers must figure out how to get paid for their content. Lowering hosting costs only solves half the equation, forcing websites to implement payment structures for the data AI extracts:

• Categorize revenue impact. Sites must separate programs that provide exposure from those that bypass human readership.
• Revoke free access. Search engines can no longer rely on business models to scrape materials without compensation.
• Establish micropayments. The market requires automatic transfers to let independent sites charge for scraping without expensive legal contracts.

Cloudflare plans to enforce this standard by blocking Google’s AI from scanning websites by default to give publishers negotiating power.

Prince argues, “if Google is just another AI company, then it should be treated like any other AI company.”

Agentic commerce demands verifiable digital trust

Beyond altering how content is monetized, AI rewrites the rules of product discovery by ignoring emotional marketing and favoring corporations with historical data. To prevent dominance of search results and allow startups to compete, small businesses must adopt digital credentials:

• Visual identities and logos hold no weight with automated buyers.
• Reputations must become machine-readable evidence that travels across platforms.
• Open standards are required to prevent a tech middleman from charging visibility fees.

Commerce now requires what Prince calls “a true attestation of quality and experience” to prove merchant reliability.