crawlers
Everything on Ground Truth tagged “crawlers” — 3 items.
Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately News
Cloudflare has made three independently configurable AI crawler categories - Search, Agent and Training - available to every customer, and from September 15 new domains will block Agent and Training traffic on ad-bearing pages by default.
Cloudflare AI crawler controls Tool
Free-tier controls that split AI crawler traffic into Search, Agent and Training, each set independently to allow, block site-wide, or block only on ad-bearing pages. These are edge blocks on classified traffic, not robots.txt requests. New domains change default on September 15, 2026.
Cloudflare AI Bot Controls Tool
A free Cloudflare setting, live since July 1 2026, that lets any site separately allow or block three kinds of AI crawler: search indexers, live AI assistants, and model-training scrapers.