web
Pew finds one in ten English web pages shows signs of AI authorship News
Pew Research sampled 490,000 English-language web pages from Common Crawl across 49 crawls between January 2021 and July 2026 and found 10% of the July 2026 sample showed significant signs of AI authorship, rising to 35% among pages that carried a publication date after ChatGPT's release.
Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately News
Cloudflare has made three independently configurable AI crawler categories - Search, Agent and Training - available to every customer, and from September 15 new domains will block Agent and Training traffic on ad-bearing pages by default.
Cloudflare Now Lets Sites Block AI Training Bots While Keeping Search News
Cloudflare has split its AI bot controls into three separately toggleable categories - search, live agent, and training crawlers - free for all sites, with training and agent bots blocked by default on ad-supported pages starting September 15, 2026.
Cloudflare AI crawler controls Tool
Free-tier controls that split AI crawler traffic into Search, Agent and Training, each set independently to allow, block site-wide, or block only on ad-bearing pages. These are edge blocks on classified traffic, not robots.txt requests. New domains change default on September 15, 2026.
Cloudflare AI Bot Controls Tool
A free Cloudflare setting, live since July 1 2026, that lets any site separately allow or block three kinds of AI crawler: search indexers, live AI assistants, and model-training scrapers.