robots.txt AI Training Opt-Out: What the Shared Token Covers

To tell AI companies not to train on your site, add a group for the user agent __usage_ai-training with Disallow: / to your robots.txt. On October 30, 2026, Anthropic, Microsoft and OpenAI said they will honor that group as a request to keep your pages out of generative AI training, and Google said it plans to read it the same way. The signal doesn't block any crawler (draft Section 4.4). Only Anthropic says it has started. Microsoft begins on February 1, 2027, Google in early 2027, and OpenAI "in the coming months."

The token comes from draft-kelley-aipref-training-preference-signal-00, an Internet-Draft written by Kevin Kelley of Anthropic, dated October 30, 2026 and expiring May 3, 2027. It builds on the IETF AI Preferences (aipref) working group's definitions, but it is an individual draft, and the datatracker page says it "is not endorsed by the IETF and has no formal standing in the IETF standards process." Use it now anyway and expect the details to move, since Google and OpenAI both describe their support as something they "aim" to deliver.

The two lines, and why User-agent: * doesn't count

The whole opt-out is one robots.txt group:

User-Agent: __usage_ai-training
Disallow: /

It works per path like any robots.txt rule. The draft's own example disallows the whole site and then allows /press-kit/, which "expresses a preference against AI training for all paths except those under /press-kit/." A disallowed path means a preference against training, an allowed path means a preference in favor, and a file with no __usage_ai-training line expresses no preference at all.

That last point catches sites that already block everything. Section 4.2 says "wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token." If your file says User-agent: * and Disallow: /, crawlers that obey it stay out, but under the new mechanism you have said nothing about training.

Screenshot: IETF Datatracker, draft-kelley-aipref-training-preference-signal-00, Sections 4.2 and 4.3. Captured November 6, 2026.

The draft also says the preference that applies to a page is the one "expressed in the robots.txt file in effect when the asset was retrieved." Adding the group now covers fetches made after each company switches the signal on. It says nothing about pages already collected, so it is no remedy for past scraping.

Readers noticed. The top comment on the r/OpenAI thread about the announcement, posted November 1 with 213 points, reads "Ah stopped scraping after already ingesting all of the internet? So kind." (Reddit). Another commenter put it shorter: "Consent doesn't work after the act" (Reddit). The draft agrees with them on the mechanics, since nothing in it asks anyone to delete what they already hold.

Who honors it, and from when

Each company also keeps its own controls, and two of them say a rule naming their crawler beats the shared token. The draft allows that: "Specific expressions SHOULD take precedence over general ones," and vendors may treat their own token as the more specific one.

Nieman Lab reported the same picture on November 2. OpenAI describes the scope as "foundational generative AI training," which is narrower wording than the draft's definition. Whether that difference matters will only show when OpenAI publishes its implementation.

https://x.com/andymstone/status/2116257949773607300

The precedence rule has a practical consequence. Anthropic says that "for a given path" a rule naming ClaudeBot wins, so on our reading a User-agent: ClaudeBot group that disallows only /private/ is what Anthropic applies, and the shared token doesn't widen it. Google's announcement promises the Google-Extended reading only when Google-Extended "is not explicitly specified," and it doesn't say what happens to a file that names it, say with Allow: / to stay in Gemini. Read your named groups before you assume the new line does anything for those two companies.

The stacking mistake that blocks Googlebot

The draft lets one group name several tokens on consecutive User-Agent lines, and "Policy tokens and other product tokens MAY be mixed in a group." Under RFC 9309, a group is "One or more user-agent lines that are followed by one or more rules." So if you paste the token line directly under an existing User-agent: * line, the wildcard and the token share whatever rules come next.

We ran three versions of a robots.txt through Python 3.12's standard urllib.robotparser and through Protego, an open-source Python parser that follows RFC 9309, and asked whether each crawler may fetch /blog/post. Both gave the same answers:

Version C turns a training preference into a crawl block on every crawler, Googlebot included. It is the easiest mistake to make when you edit a live file by hand, because the result looks like one tidy group. Give the token its own group with its own Disallow line, separated from the others by a blank line.

What the token leaves out

Section 4.4 says "This policy token is not a directive for crawling behavior." A company "MAY" use your preference to skip fetching pages it would only use for training, but nothing in the draft requires it, and the draft tells you to name crawlers if you want to "limit crawling."

It also covers less than "AI." Training here means "Using an asset to modify the learned parameters of a generative AI model," and the definition excludes models used "for classification, ranking, or scoring." Search indexing, ranking and grounding (fetching your page to answer a question in a chatbot) are separate uses.

Grounding is where Google and Microsoft differ. Google's docs say Google-Extended controls both training of "future generations of Gemini models" and grounding in Gemini Apps, and its blog adds that "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Since Google will read the shared token as a Google-Extended disallow, we read the two together as one switch, so opting out of Gemini training also takes you out of Gemini Apps grounding. Google's announcement doesn't describe a setting for one without the other.

Microsoft splits them. Its grounding preference is page-level only ("There is no site-wide robots.txt grounding preference"), with DisallowExtended to allow grounding in Microsoft's own products but not in grounding services it provides to third parties, such as Web IQ.

Finally, the draft says "Private agreements MAY override preferences expressed via this mechanism," and Microsoft says its settings "do not establish licensing or payment arrangements." If you have signed a content deal, the contract can override what your robots.txt says.

Bing users: the March 1, 2027 deadline

If your site uses NoArchive or NoCache to keep pages out of Microsoft's AI products, those tags stop counting. Microsoft's post says "after that date, NoArchive and NoCache will no longer be read as a preference for training or grounding," and the date is March 1, 2027. The new settings take effect "as it crawls" from February 1, 2027.

Screenshot: Bing Blog, "Your content, your choice: clearer preferences for training and grounding," Microsoft. Captured November 6, 2026.

For training, use the robots.txt group above, or a page-level tag:

<meta name="Bingbot" content="ai-training:disallow">
X-Robots-Tag: Bingbot: ai-training:disallow

For grounding, use ai-grounding:disallow or ai-grounding:disallowextended in the same meta tag or header. Sites that used NoArchive or NoCache as of December 15, 2026 can also record a temporary grounding preference in Bing Webmaster Tools from that date, which holds until the page is recrawled or September 30, 2027, whichever comes first. There is "no temporary training setting in Bing Webmaster Tools," so the training line has to be live on your site. Microsoft adds that "Publishing a setting is not the same as activating it," since Bing has to recrawl the page to see it.

Nobody had added it yet: 24 robots.txt files on November 6

We fetched robots.txt from 30 large sites on the evening of November 6, 2026 (UTC); 24 returned a usable file. None of the 24 had a __usage_ai-training group, including anthropic.com, google.com, microsoft.com and openai.com. Many already name the vendors' crawlers one by one:

The 24 were news, reference and platform sites (nytimes.com, apnews.com, ft.com, npr.org, nature.com, reddit.com, wikipedia.org, github.com and others; the full list is in our source log). Six didn't count: two timed out, three refused the request, and theguardian.com served a bot-check page. Even anthropic.com, whose engineer wrote the draft, had only a User-Agent: * group with Allow: /. For sites like nytimes.com that already name GPTBot, ClaudeBot and Google-Extended, the shared token changes little at Anthropic and Google, because the named groups take precedence. It matters most for a site that names nobody, and for crawlers that adopt the token later.

Steps for a small site this week

1. Open your live robots.txt at yoursite.com/robots.txt, not the copy in your CMS or repo. Note every group that names an AI crawler.
2. Add the token as its own group at the end of the file, after a blank line: User-Agent: __usage_ai-training then Disallow: /. Never stack it under User-agent: *. Add Allow: lines for paths you want trained on, such as a press kit.
3. Keep your named groups. GPTBot, ClaudeBot and Google-Extended rules still decide at those companies, and OpenAI and Google haven't switched the token on yet. Meta hasn't committed, so keep or add User-agent: Meta-ExternalAgent with Disallow: / if you don't want Meta crawling for AI (our Meta terms post explains why that block still matters).
4. Test before you publish. Check that Googlebot and Bingbot can still fetch your main pages, and that any Allow: path comes out allowed for __usage_ai-training. Use a parser that applies RFC 9309's longest match, such as Protego. Python 3.12's urllib.robotparser catches version C above, but it applies rules in file order, so for the draft's press-kit example (Disallow: / then Allow: /press-kit/) it reported /press-kit/ blocked where Protego reported it allowed.
5. If you use NoArchive or NoCache, add the Bingbot ai-training and ai-grounding settings before March 1, 2027, and record the temporary grounding preference in Bing Webmaster Tools once it opens on December 15.
6. Keep a dated copy of each robots.txt version. The draft ties the preference to the file "in effect when the asset was retrieved," so if you ever need to show what you asked for and when, the copy is your record.

If you run AI tools on other people's content rather than publishing your own, the same file now tells you two things per site, whether you may crawl and whether the owner objects to training. Our post on Like Company v Google covers why a log of what you fetched and under which robots.txt matters there. Providers working to the EU code should also note that Google says the proposal "is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers," which already points to robots.txt; our GPAI Code of Practice guide has the text.

What we could not confirm

We couldn't confirm when OpenAI will switch the signal on, or the exact date in "early 2027" for Google. Anthropic says it is rolling out "over time," and its page doesn't say which services already apply the token. Meta's support was a statement on X quoted by AFP and Nieman Lab, and its crawler documentation doesn't mention the token. We tested two Python parsers and did not run Google's or Bing's own. The draft is version 00 of an individual submission, so the token name and rules could change before any working group adopts it.

Related Reading

• Meta Commercial Terms Paused: AI Crawl Clause and What to Do
• Like Company v Google (C-250/25): When RAG Chatbots Infringe
• EU AI Act GPAI Code of Practice: What It Requires
• AI Companies Built Empires by Scraping the Internet. Now They're Suing Anyone Who Scrapes Them Back.
• USA Today Sues OpenAI: $250M Claim Over ChatGPT News Summaries