News · 2026-10-31

OpenAI plans to honor a shared website signal for AI training preferences

OpenAI said on October 30 that it plans to honor a shared robots.txt signal through which websites can express preferences about foundation-model training. The announcement could reduce the need for separate provider-specific instructions, but it describes a future implementation and does not create a universal block on scraping or promise to undo past training.

Key facts

• OpenAI announced the plan on October 30 and described implementation as coming “in the coming months.”
• The proposed common signal lives in robots.txt, the file websites already use to communicate crawler rules.
• Existing OpenAI-specific crawler controls will continue to apply.
• The primary source is OpenAI’s training-preference announcement, which links to ongoing internet standards work.

For a publisher, the problem is partly administrative. A site may want search engines to index its pages while objecting to their use in training a generative model. When each provider expects a different bot-specific instruction, maintaining that choice requires tracking a growing collection of names and behaviors.

A shared signal is intended to separate the intended use from the identity of a particular crawler. Think of a library notice that distinguishes permission to consult a book from permission to use its contents for a separate project. The notice can communicate the distinction consistently. It still depends on the recipient reading it and following the stated preference.

OpenAI’s announcement concerns foundational generative-AI training. It says the company will honor a common robots.txt signal, but it does not specify which final syntax it will adopt. The timing phrase is a commitment to future implementation, not evidence that a website can already deploy the new control and expect OpenAI to apply it.

The linked standards documents help explain both the promise and the uncertainty. An individual Internet-Draft proposes a robots.txt group keyed by User-Agent: __usage_ai-training, using ordinary Allow and Disallow paths. That design treats the training preference separately from whether a bot may fetch a page at all.

A separate working-group attachment draft describes Content-Usage mechanisms through both an HTTP header and a robots.txt directive. These are different proposals in an evolving process. OpenAI has not said that the individual draft’s exact syntax, the header mechanism, or a particular combination is its final implementation. The vocabulary draft also records the work’s provisional status. An Internet-Draft is not an approved internet standard.

This distinction prevents a tempting but consequential error: publishing a definitive configuration recipe before the company has published its own details. A technically plausible setting from a draft is still a draft setting. The verified news is OpenAI’s stated intention to participate, with implementation specifics to follow.

The existing controls are more concrete: OpenAI distinguishes two crawler roles, one for training-related collection and one for search. This distinction gives publishers a useful current choice: they can express a preference against training while still allowing search access, rather than treating every use of a public page as a single permission. Its crawler documentation says disallowing GPTBot communicates that crawled content should not be used to train its generative-AI foundation models. OAI-SearchBot is a separate crawler for ChatGPT search. A site can therefore express different choices for training and search today using the documented provider-specific controls. The new shared preference is described as additive to those rules.

The most important limit is that a preference signal is not an enforcement barrier. It does not prevent a nonparticipating scraper from fetching a public page, force all AI companies to comply, or guarantee that content disappears from previously collected datasets. OpenAI does not promise removal of material already used in training, and its handling of every future preference change remains unspecified.

The broader debate also includes whether aggressive anti-bot measures burden legitimate access without stopping bad actors. The Tech Policy Press essay on anti-stealth-crawler proposals argues that restrictive approaches can damage useful scraping while leaving abusive behavior unresolved. That is relevant policy criticism, not a test of OpenAI’s new signal. No final cross-industry consensus or measured compliance result is established by the documents reviewed here.

For publishers, the next practical milestone is an implementation notice precise enough to test: accepted syntax, effective date, participating providers, and rules for changes over time. For AI users, the story concerns how training inputs acquire usable governance instructions, a question connected to dataset documentation and training-data attribution. A common vocabulary can make preferences easier to communicate. Whether it changes collection and training behavior will depend on implementation and compliance.

Primary source, verified: read the paper →

Ground Truth. (2026, October 31). OpenAI plans to honor a shared website signal for AI training preferences. Ground Truth. https://groundtruth.day/news/openai-shared-training-preference-robots.html

@misc{groundtruth:openai-shared-training-preference-robots,
  title  = {OpenAI plans to honor a shared website signal for AI training preferences},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/openai-shared-training-preference-robots.html}
}

Topics: openai · training-data · web · standards · publishers