Cloudflare, a US-based cloud connectivity and security company, launched Disallow AI Training on September 16, a setting that lets websites block AI training while remaining indexed for search.

The feature is part of Cloudflare’s AI crawl controls. It works with Bot Preference Sync, which lets site owners set crawler preferences once across supported crawlers.

Cloudflare defined mixed-use crawlers as bots that collect content for both search indexing and AI training. The company said this can force publishers to choose between staying visible in search results and blocking AI training.

Cloudflare also introduced an Accountable designation for AI crawling. It said Apple, Google, and Microsoft now carry the label because they met the criteria or gave timelines to meet them.

The criteria include AI training opt-outs, AI-generated summary opt-outs, URL-level visibility, and confirmation that training opt-outs will not affect search rankings.

Cloudflare said mixed-use crawlers make up 36.6% of verified crawler traffic on its network. It added that 17% of websites restrict AI training, while fewer than 1% block search crawlers.

🔗 Source: Cloudflare

Recent Cloudflare developments