Google, Microsoft and Anthropic Back One robots.txt Line to Opt Out of AI Training

TL;DR

On October 30, Google and Microsoft said they will honor a shared AI training opt-out in robots.txt, and an Anthropic engineer published the IETF draft that defines it. The syntax is an ordinary rule group under a reserved name, User-Agent: __usage_ai-training. Bing starts reading it on February 1, 2027. Google targets 2027 and will treat it as a Google-Extended opt-out. Anthropic says it has started implementing it. Microsoft also adds a separate per-page switch for AI answers, so you can refuse training and still get cited.

What was announced

Three things landed within a few hours of each other.

• The spec. Kevin Kelley, listed as Anthropic on the draft, submitted draft-kelley-aipref-training-preference-signal-00 to the IETF AI Preferences (AIPREF) working group. In his post to the working group list he wrote that "Anthropic has already started implementing support for this draft."
• Google. In a Search Central post signed by John Mueller, Google says it plans to support the shared control and aims to do so "along with other AI providers in 2027."
• Microsoft. Bing's post, by principal product manager Krishna Madhavan, gives the exact syntax, a start date of February 1, 2027, and a migration deadline for sites that relied on the old NoArchive and NoCache tags.

Search Engine Roundtable reports that OpenAI announced support as well. We could not load OpenAI's page to check its wording. The draft's acknowledgments do thank two people listed with OpenAI affiliations, along with participants from Google, Microsoft and Anthropic.

Until now, opting out of training meant keeping a list of vendor tokens: Google-Extended for Google, GPTBot for OpenAI, and so on. Each one also had to be checked against whatever else that crawler did for you. The new token is meant to replace that list with one rule group that every participating provider reads.

The syntax

The draft adds no new directive and no new grammar. It borrows the existing User-Agent line from RFC 9309 and puts a reserved "policy token" in it where a crawler name would normally go. This is the whole-site opt-out from Bing's guide:

User-Agent: __usage_ai-training
Disallow: /

Path rules work the way they always have, including longest-match precedence. Kelley's email gives this example, which opts out of training everywhere except a press kit you want models to learn:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Think of the policy token as a fake crawler that never visits. Nobody fetches anything as __usage_ai-training. The group is a sign on the door that every participating crawler agrees to read, while each crawler's own group still decides whether it can come in at all. The draft spells this out: the token "is not a directive for crawling behavior," and crawl rules for Googlebot or Bingbot keep the meaning they have today. A crawler that collects only for training "MAY" choose not to fetch pages it can't use.

Three rules that will trip people up

• The wildcard does not count. The draft says User-Agent: * groups "MUST NOT be interpreted as matching the policy token." A site that blocks everything under * has expressed no training preference under this mechanism. That was deliberate, Kelley writes, so that no robots.txt file published before today accidentally says anything.
• No group means no preference. If the file has no __usage_ai-training line, nothing has been expressed. Bing says outright that both of its new preferences default to Allow.
• Specific beats general, and contracts beat both. Providers may treat their own tokens, such as Google-Extended, as more specific than the shared one. Under the draft, "Private agreements MAY override preferences expressed via this mechanism." If you have signed a licensing deal, the deal decides.

Who honors it, and when

Google: an alias for Google-Extended

Google is not adding new behavior. If your robots.txt opts out through the policy token "and Google-Extended is not explicitly specified," Google will treat it "as if the publisher had disallowed Google-Extended," the token it has offered since 2023. If you do name Google-Extended, that rule wins. The post also notes that the scheme is consistent with the EU's General-Purpose AI Code of Practice, which recognizes robots.txt as a primary way to express machine-readable preferences.

One sentence in Google's post matters for traffic: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Google doesn't say what happens to grounding when the shared token turns Google-Extended off. Our reading is that, for Google, one opt-out covers training and Gemini Apps grounding together. That is an inference, and Google hasn't confirmed it.

Microsoft: training and answers split apart

Bing goes further. Training can be set site-wide in robots.txt and overridden on individual pages with a meta tag or header:

<meta name="Bingbot" content="ai-training:disallow">
X-Robots-Tag: Bingbot: ai-training:disallow

Grounding, meaning the use of your page to inform a generated answer, is a separate, page-only preference (ai-grounding:) with three values:

• Allow (the default): your content can inform answers anywhere Microsoft serves them.
• DisallowExtended: only Microsoft-operated experiences such as Copilot and Microsoft 365, excluding grounding Microsoft sells to third parties through Web IQ and the grounding API in Foundry.
• Disallow: no grounding at all, at a cost Microsoft states plainly: "you will see reduced brand visibility in AI experiences."

None of these settings affects eligibility for regular Bing search results. Microsoft is equally plain about the other side: "Neither choice guarantees inclusion, traffic, or payment." A preference is not a license, and it doesn't come with a check.

The Bing migration has a deadline

If your site uses NoArchive or NoCache to keep content out of Bing's AI, those tags are being retired as AI signals. Bing will ignore them for training and grounding after March 1, 2027. Here is the schedule:

Bing only learns your new settings when it recrawls a page, and that might not happen between February 1 and March 1. To cover the gap, sites still using the old tags on December 15 can record a temporary grounding preference in Bing Webmaster Tools. That temporary setting lasts until the page is recrawled or until September 30, 2027, whichever comes first. If you miss the March 1 deadline, pages that used NoArchive or NoCache fall back to Allow for grounding until Bing recrawls them. There is no stopgap for training: the robots.txt or page-level preference has to be live by March 1. Microsoft's FAQ covers the edge cases.

What is still unsettled

• It is an individual draft. Kelley's document is a personal submission meant as input to the working group's attachment work, which the group set aside earlier so it could finish the vocabulary first. Version -00 expires on May 3, 2027. The definition of "AI Training" comes from draft-ietf-aipref-vocab-08, which is also still a draft.
• Only training is covered. Google says the shared mechanism carries "the AI training definition" while "multistakeholder discussions continue on the rest of the AIPREF vocabulary." Search, AI answers and AI agents have no shared token yet.
• It is voluntary. Kelley's email says the interpretation "may change" as standards evolve. Robots.txt has always depended on the crawler's good manners, and so does this. A scraper that ignores Disallow today will ignore this too.

What to do with your robots.txt

If you want to opt out of training broadly, add the __usage_ai-training group now. RFC 9309 parsers that don't know the token should skip it without any effect on your other rules, which is the point of reusing the existing syntax. Keep your vendor-specific lines as well: Google-Extended and the others still work and take precedence where a provider says so. For Bing, plan the page-level grounding decision separately. For many publishers the useful setup is no training with grounding allowed, so you keep the citations without donating the corpus. Small sites and SaaS docs teams have one sensible question to answer: do you want AI answers quoting your docs, or do you want out entirely?

Key Takeaways

• A new IETF draft by Anthropic's Kevin Kelley defines User-Agent: __usage_ai-training as a shared robots.txt group for AI training preferences, using only existing RFC 9309 syntax.
• Microsoft will honor it from February 1, 2027. Google aims for 2027 and treats an opt-out as disallowing Google-Extended. Anthropic says it has started implementing it.
• User-Agent: * never matches the token, and a file without the token expresses no preference. Existing blanket blocks do not count as a training opt-out.
• Bing adds a separate per-page grounding switch (Allow, DisallowExtended, Disallow), so a site can refuse training and still appear in Copilot answers.
• Sites using NoArchive or NoCache for AI control must move to the new preferences by March 1, 2027, or Bing will stop treating those tags as AI signals.
• The signal covers training only, is voluntary, and private licensing deals override it.

Sources: Google Search Central Blog, Microsoft Bing Blog, Bing Webmaster Tools FAQ, IETF draft-kelley-aipref-training-preference-signal-00, Kevin Kelley on the ai-control list, draft-ietf-aipref-vocab-08, RFC 9309, Search Engine Roundtable (Google), Search Engine Roundtable (Bing)