OpenAI announced on October 30, 2026 that it will honor a common robots.txt signal for foundational generative AI training preferences. Its announcement frames the voluntary effort as a way for website operators to communicate one preference to multiple participating providers. Implementation is planned for the coming months; the announcement does not say the new behavior is already deployed.
What the shared signal expresses
The linked Internet-Draft proposes the policy token __usage_ai-training inside ordinary robots.txt groups. It uses existing path rules to express a preference for or against training use. A wildcard user-agent group does not implicitly match that policy token.
The proposal distinguishes content-use preferences from crawler traffic rules. It also remains a draft within an ongoing standards discussion. Calling this a completed universal standard would get ahead of both the document's status and provider implementation. OpenAI explicitly says its existing crawler controls will continue to apply.
Search discovery remains a separate choice
OpenAI's crawler documentation already separates OAI-SearchBot from GPTBot. It describes the former as serving search discovery and the latter as crawling content that may train foundation models. A publisher can allow search indexing while expressing a different choice about training.
That distinction is useful for a business publishing product documentation. The company may want prospective customers to find a public integration guide, while reserving a separate policy for training use. A single broad rule made without considering those purposes can create a different discoverability outcome from the one the content owner intended.
A preference file is a public policy surface
RFC 9309, the underlying Robots Exclusion Protocol, explains that robots rules are not access authorization. The file cannot protect a private customer document. Authentication and resource permissions still belong at the application or hosting boundary.
For publishers, the practical work starts with ownership. Decide who can approve the training preference, who maintains the deployed file, and which public paths need exceptions. Keep that decision beside the existing search and crawler policy so a routine search optimization does not accidentally overwrite a separately approved training choice.
How to prepare without assuming rollout
A useful preparation exercise is to map a few representative URLs: a public article, a product page, a downloadable report and a login-protected resource. For each one, write down the desired search visibility and training preference independently. The protected resource should stay protected regardless of the robots configuration.
Then inspect the file actually served by each relevant hostname. A content platform, reverse proxy and marketing site can have different release paths, so reviewing a repository copy alone may miss the live output. Record the approved configuration and verify that ordinary publishing updates preserve it.
Finally, distinguish policy intent from evidence of provider behavior. An operator can demonstrate what its file says today; that does not demonstrate that every provider has implemented or honored the signal. Review provider implementation announcements as they arrive. The immediate benefit of the October commitment is a clearer route toward shared expression, with a concrete migration question for site owners once implementation details are published.