Google, Microsoft and OpenAI have announced plans to honour a single shared robots.txt line that lets a website state its AI training preference once, instead of writing separate rules for each company’s web crawler. The move, reported on 1 November, could simplify life for publishers who want to keep their content out of generative AI training.
At present, a site that wants to refuse AI training must write different rules for different crawlers, such as Google-Extended, GPTBot and others. Under the new approach, a site would write one ordinary rule group under a policy token called __usage_ai-training, using the familiar Allow and Disallow lines. An example in the proposal blocks an entire site except for its press-kit folder.
The line comes from an Internet Draft submitted on 30 October by Kevin Kelley of Anthropic to the AI Preferences working group at the Internet Engineering Task Force (IETF). The draft is listed as an individual submission, which means it does not represent official IETF standardisation activity, and it is set to expire on 3 May 2027 unless a new version is published.
How the shared signal works
The draft adds no new directive to robots.txt. It reuses the file’s existing structure, but instead of naming a crawler on the User-Agent line, it names the policy token. The draft defines AI training as using content to modify a generative model’s learned parameters, and it says models that only do non-generative work, such as classification or ranking, are exempt.
There are important limits. A wildcard group, written as User-Agent with an asterisk, can never match the token, and a robots.txt file with no __usage_ai-training group signals no preference at all. The token is also a statement about how content may be used, not an instruction about crawling itself, so it does not stop a crawler from visiting a site.
Different timelines for each company
Based on the timelines the companies gave, none of the three currently supports the shared line yet. Google aims to support it in early 2027, and says that where a site opts out through the shared line without specifying Google-Extended, it will treat that as if the publisher had disallowed Google-Extended, a control covering future Gemini model training and grounding in Gemini Apps and on Vertex AI.
Microsoft will begin honouring the new options as it crawls from 1 February 2027, and will stop recognising its NoArchive and NoCache tags as training or grounding signals after 1 March 2027. It is also adding a separate page-level AI Grounding preference, which can be set by a meta tag or an HTTP header, with a DisallowExtended option that limits grounding to Microsoft’s own tools.
OpenAI aims to honour the signal within the next few months, while its current robots.txt policy stays in effect. Anthropic, whose engineer wrote the proposal, has said it has already started building support. Microsoft has also warned that blocking AI grounding will significantly reduce a site’s visibility in AI-generated experiences, and that the new preferences do not establish licensing or payment arrangements.
For publishers, the appeal is simplicity: one line, recognised by several companies, instead of a patchwork of crawler-specific rules. The open question is how consistently the companies will interpret that line once support actually arrives.
