web
Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately News
Cloudflare has made three independently configurable AI crawler categories - Search, Agent and Training - available to every customer, and from September 15 new domains will block Agent and Training traffic on ad-bearing pages by default.
Cloudflare Now Lets Sites Block AI Training Bots While Keeping Search News
Cloudflare has split its AI bot controls into three separately toggleable categories - search, live agent, and training crawlers - free for all sites, with training and agent bots blocked by default on ad-supported pages starting September 15, 2026.
Cloudflare AI crawler controls Tool
Free-tier controls that split AI crawler traffic into Search, Agent and Training, each set independently to allow, block site-wide, or block only on ad-bearing pages. These are edge blocks on classified traffic, not robots.txt requests. New domains change default on September 15, 2026.
Cloudflare AI Bot Controls Tool
A free Cloudflare setting, live since July 1 2026, that lets any site separately allow or block three kinds of AI crawler: search indexers, live AI assistants, and model-training scrapers.