News · 2026-07-25

Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately

Cloudflare has split AI crawler traffic into three independently configurable categories - Search, Agent and Training - and made the controls available to every customer, including on the free tier. A site can now allow the crawler that indexes it for search while blocking the crawler that scrapes it for training data, which was not cleanly possible before. From September 15, 2026, new domains will default to blocking Agent and Training traffic on pages Cloudflare detects as carrying ads, while continuing to allow Search.

Key facts

• Three categories, each independently set to allow, block site-wide, or block on ad-bearing pages, per Cloudflare's bot documentation.
• September 15, 2026: new domains default to blocking Agent and Training on ad-bearing pages.
• Mixed-purpose crawlers that combine Search and Training are blocked by every training-blocking configuration, including the legacy "Block AI bots" toggle.
• Existing customers can configure the policy now or opt out of the default change; the controls are free.

What the three categories actually mean

Cloudflare's definitions are the operative text, and they are more precise than the debate around them. Search is a crawler that collects or indexes your content in order to answer questions about it later. Agent is automated activity acting in real time on a person's behalf - a chat assistant fetching a page because a user asked about it, or a browser-use agent clicking through a site. Training is a crawler taking your content to train or fine-tune a model.

The reason those three needed separating is that they have different bargains attached. Search has always traded indexing for traffic. Agent traffic arguably delivers a reader, just not one who sees your page. Training delivers nothing back at all. Bundling them under one "AI bots" switch forced publishers into an all-or-nothing choice that matched none of their actual preferences.

Crucially, these are edge policies applied to classified traffic - both verified bots and relevant unverified ones - not robots.txt preferences that a crawler may or may not respect. A blocked request does not get served.

The Googlebot problem

The sharpest issue surfaced immediately in the Hacker News discussion: what happens to crawlers that do both jobs? Cloudflare's answer is explicit. Mixed-purpose crawlers - ones classified as serving both Search and Training - are blocked by all configurations that block training, including the older blanket option. A site that decides it does not want its content training models may therefore also block Googlebot, Applebot or Bingbot where Cloudflare classifies them as dual-purpose.

That is not a bug in the design, it is the design surfacing an uncomfortable fact: the large search crawlers stopped being purely search crawlers some time ago, and publishers have not had to price that until now. Deciding whether to accept training in exchange for search visibility is now an explicit, configurable choice with a visible cost on both sides.

Where enforcement stops

The honest limit is one Cloudflare states itself. Alongside the blocking controls, it manages a robots.txt content signal with values covering immediate use, reference, and full reuse. Those are preferences, not enforceable blocks. More granular controls tied to what a crawler does downstream are still described as in development.

The reason is structural rather than a shortcoming of the product. Edge blocking governs identifiable traffic at the moment it arrives. It cannot inspect what a crawler later does with content it was allowed to fetch, and it cannot govern content that reaches a model through a third party. Enforcement ends where verifiable traffic behavior ends. The skeptical "this is still an honor system" framing is wrong about the blocks and right about the signals.

Why it matters for open models too

This is the other half of a story usually told only from the model side. The same week that llama.cpp turned local inference servers into tool hosts, the web's largest reverse proxy gave sites a purpose-based policy layer for inbound agents. Open weights need a runtime and tool layer inside the stack; the open web needs a traffic and policy layer at its perimeter. Both arrived at once, and both are about control over deployment rather than model capability.

For anyone running a site, the practical read is short: the default is changing on September 15 for new domains, mixed-purpose crawler classification is the setting that will surprise people, and the controls cost nothing to configure now.

The honest caveat

Classification is doing enormous work here. Whether a given crawler lands in Search, Agent, Training or two of them at once is Cloudflare's judgment, applied at scale, to traffic that has every incentive to be ambiguous. The categories are only as good as that classification, and Cloudflare has not published how contested cases are resolved or how a crawler operator appeals a label.

Primary source, verified: read the paper →

Ground Truth. (2026, July 25). Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately. Ground Truth. https://groundtruth.day/news/cloudflare-splits-ai-crawlers-into-search-agent-and-training.html

@misc{groundtruth:cloudflare-splits-ai-crawlers-into-search-agent-and-training,
  title  = {Cloudflare now lets any site allow search crawlers while blocking AI agents and training bots separately},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/cloudflare-splits-ai-crawlers-into-search-agent-and-training.html}
}

Topics: cloudflare · web · crawlers · policy · publishing · infrastructure