For years, telling AI companies not to train on your website meant a small zoo of crawler rules — Google-Extended, GPTBot, ClaudeBot, CCBot — each with its own syntax and no guarantee the next model would respect any of them.
That changed on October 30, 2026, when Google, Microsoft, OpenAI and Anthropic each announced they will honour one shared AI training opt-out in robots.txt: a policy token called __usage_ai-training. One small group in your robots.txt becomes a binding instruction — as each company’s rollout lands — not to use your content to train generative AI models.
Here is what happened, how the token works, where the four timelines diverge, and what to do now — whether your audience is in London, Dubai or Riyadh.
What Happened on October 30, 2026
Four announcements, one mechanism:
- Google published “Supporting a shared AI training control for websites” on the Google Search Central Blog (by Search Advocate John Mueller) and added a note to its crawler docs the same day.
- Microsoft published “Your content, your choice” on the Bing Blog — adding a separate AI grounding setting and a deadline for its older controls.
- OpenAI published “Making it easier for websites to express AI training preferences”, committing to honour the shared signal.
- Anthropic moved first in practice: Kevin Kelley submitted the specification to the IETF’s AI Preferences (AIPREF) working group that afternoon — Internet-Draft draft-kelley-aipref-training-preference-signal-00 — and said Anthropic had already started implementing it in ClaudeBot.
One caveat: the specification is an individual Internet-Draft — the lowest rung of the IETF ladder, with no working-group endorsement and no formal standing. It is intended for the Standards Track, expires on May 3, 2027 unless revised, and binds only the companies that choose to follow it. The commitments are real; the standardisation is not finished.
How the Shared AI Training Opt-Out robots.txt Token Works
The syntax is deliberately boring — ordinary robots.txt, with one twist:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
The twist is that __usage_ai-training names a use, not a crawler. It is a “policy token”: it sits where a crawler name would sit on the User-Agent line, but no crawler ever sends it. Instead of saying “Google-Extended, stay out”, you are saying “whoever you are, if you train models, this path is off limits”. The double-underscore prefix marks a policy-namespace token rather than a literal bot name under RFC 9309.
Three properties make it practical. It is not a crawl directive — the draft states the token “is not a directive for crawling behavior”. Crawlers keep fetching normally; they simply must not use the content for training, so search traffic is untouched. It composes with normal path rules: Allow lines carve out exceptions like a press kit you want in training data. It is safe to add today: software that does not implement the token ignores the group exactly as it ignores any unrecognised User-Agent.
Why One AI Training Opt-Out Line Changes Everything
Until now, opting out was an N-by-M problem: N crawlers, M syntaxes, and most site owners simply never bothered — which is exactly why training-data consent has been such a mess. One shared signal collapses that to a single decision: a few lines of plain text in a file you already own, no vendor dashboard, no legal agreement. For publishers, AI training consent goes from specialist chore to checkbox — and it is the first time the four largest Western AI labs have lined up behind one opt-out mechanism.
It also settles a question that has dogged generative engine optimisation: training data and search visibility are separate systems. If you are working to get cited in AI Overviews, that work is untouched by this token — our guide on how to get cited in AI Overviews covers the content patterns that earn citations, and none of them change here.
What Each Provider Committed — the Timelines Are Not the Same
“All four agreed” is true; “all four switch on together” is not. The rollouts run on four different schedules, and Microsoft’s announcement goes further than the rest. PPC Land has a detailed rundown of all four announcements with the timelines side by side.
Microsoft: February 1, 2027 — and a Hard Deadline for noarchive
Microsoft gave the firmest commitment: Bing honours the new preferences from February 1, 2027, as Bingbot crawls each page. It also added a separate AI Grounding preference — whether Microsoft’s AI answers may draw on a page — via a page-level Bingbot meta tag, and retired the old controls: Bing has read noarchive and nocache as AI signals since 2023, but after March 1, 2027 they stop counting, with a Webmaster Tools transition opening December 15, 2026. If your site relies on those tags, that setup has an expiry date — migrate deliberately, not in March.
Google: Early 2027 — Mapped to Google-Extended
Google’s blog post aims for support “along with other AI providers in 2027”; the note added to its crawler documentation says early 2027. The key detail is the mapping: when a site opts out of training with the shared token and Google-Extended is not explicitly specified, Google plans to treat that as a Google-Extended disallow. Explicit Google-Extended rules keep working alongside it and take precedence where set.
Google-Extended does not affect your inclusion in Google Search, nor is it a ranking signal. It covers Gemini model training and grounding — Search-index content handed to the model at prompt time in Gemini Apps and Grounding with Google Search on Vertex AI. Want your content eligible in Gemini Apps responses? Explicitly allow Google-Extended.
Critically, the token does nothing to AI Overviews or AI Mode — those have their own site-wide switch in Search Console under Settings, then Search generative AI. Google’s October AI Overviews update shows how fast that surface moves; manage it separately.
OpenAI: “In the Coming Months”
OpenAI committed without publishing a date, promising an update when support ships. For OpenAI traffic, the shared token is a promise, not yet a mechanism — keep your existing crawler rules.
Anthropic: Already Rolling Out
Anthropic has already started implementing support — ClaudeBot first — with wider rollout taking “the coming months”. Kelley warned to expect “complications and edge cases”: this is the mechanism’s first real-world deployment.
The Google-Extended Decision UK, UAE and Saudi Businesses Must Make
- You already disallow Google-Extended and want to keep it that way. Add the shared token for future-proofing across Microsoft, OpenAI and Anthropic. For Google, nothing changes: your explicit rule stands.
- You allow Google-Extended because you want visibility in Gemini Apps. Do not add a blanket Disallow under the shared token without thinking — Google will read it as a Google-Extended disallow unless Google-Extended is explicitly specified. Scope your disallow to the paths you actually want out of training.
- You have never set any AI controls. The shared token is the simplest first step you have ever had: one group, safe to add now, effective as each provider’s rollout lands through 2027.
One note on our markets: GSC shows our UAE and Saudi audiences search in English, and English content trains the same models wherever the reader sits — this applies to a Gulf business exactly as to a UK publisher. The more precisely Google understands your business as an entity, the more deliberate you can be about which content represents you; our entity SEO explainer covers that foundation.
What to Do Now: Your AI Training Opt-Out Checklist
- Add the group to robots.txt today. Inert for non-implementers, already live for ClaudeBot — the SEO Handbook’s practical walkthrough covers the exact syntax and edge cases.
- Keep your existing per-crawler rules until each provider’s rollout date passes — removing them early buys you nothing.
- Decide your stance as a business. This is an IP and competitive-positioning call: do you want your content shaping competitors’ models, or out? Make it once, document it, apply it consistently.
- If you depend on noarchive or nocache for Bing, plan the migration. The March 1, 2027 cutoff is fixed; the Webmaster Tools transition opens December 15, 2026.
- Do not conflate the controls. Training opt-out, AI Overviews visibility and search crawling are three separate switches — audit each deliberately.
Frequently Asked Questions
Will adding the __usage_ai-training token hurt my Google rankings?
No. The token states a training preference; it is not a crawl directive and carries no ranking signal. Google’s own documentation says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” — the shared token inherits that property.
Does this stop my content appearing in AI Overviews or AI Mode?
No. Those surfaces are governed by a separate Search Console setting (Settings, then Search generative AI), not by robots.txt training controls.
Should I remove my existing AI crawler rules like Google-Extended?
Not yet. The token takes effect only as each provider rolls out — Google early 2027, Microsoft February 1, 2027, OpenAI in the coming months. Keep existing rules until your traffic is covered, then consolidate.
Is this an official web standard?
No. It is an individual IETF Internet-Draft with no formal standing in the IETF process, intended for the Standards Track and expiring May 3, 2027 unless revised. The vendor commitments are real; the standardisation is not finished.
One Line Today, Control Through 2027
At Perspective Media, Semantic SEO — AEO, GEO and AI-search visibility — is one of our core services. We help businesses across the UK, UAE and Saudi Arabia stay visible, and in control, as search goes AI-first. Start with our generative engine optimisation guide — or get in touch for a robots.txt and crawler-controls audit.
