publishing
Everything on Ground Truth tagged “publishing” — 2 items.
Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately News
Cloudflare has made three independently configurable AI crawler categories - Search, Agent and Training - available to every customer, and from September 15 new domains will block Agent and Training traffic on ad-bearing pages by default.
Cloudflare AI crawler controls Tool
Free-tier controls that split AI crawler traffic into Search, Agent and Training, each set independently to allow, block site-wide, or block only on ad-bearing pages. These are edge blocks on classified traffic, not robots.txt requests. New domains change default on September 15, 2026.