OpenAI, Google, Microsoft and Anthropic to adopt shared AI scraping opt-out
OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI. The news was first reported by Search Engine Roundtable and then covered by The Verge. The point of the control is that publishers can set their AI training preferences once and have them apply across multiple AI companies, instead of configuring each company separately, which is what sites have to do today.
The signal is a shared "policy token". It was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live inside a website's robots.txt file, the long-standing mechanism that tells web crawlers what they can and cannot scrape, whether for search indexing, archiving or AI training.
The four companies say that if their AI crawlers encounter the shared token on a site, they will restrict themselves from accessing that site. Timing differs by company. Anthropic has already adopted the signal. Google and Microsoft plan to support it next year. OpenAI says it will implement it in the "coming months."
Meta is a partial case. Communications head Andy Stone signaled support for the new setting, but the report says he does not say whether Meta will honor it. His statement, added in an October 30th update to The Verge's piece: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry."
Microsoft adds that it will offer granular controls letting publishers opt out of AI training or of AI grounding, which is the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing's search results regardless of their preferences. Google, for its part, previously introduced similar controls that let sites decide whether their content appears in AI search features or is used for AI training, without affecting visibility in search.
The announcement lands as publishers deal with a continuing flood of AI crawlers scraping their sites and feeding the content into AI models. Other efforts against unauthorized scraping include the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers that visit their sites, and Cloudflare's default blocking of AI crawlers.
Key facts
- OpenAI, Google, Microsoft and Anthropic say they will adopt a shared "policy token" signal that lets publishers set AI training preferences once for multiple AI companies.
- The token was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the IETF, and will live in a site's robots.txt file.
- The four companies say they will restrict their AI crawlers from a site if they encounter the token.
- Anthropic has already adopted the signal; Google and Microsoft plan to support it next year; OpenAI says it will do so in the "coming months."
- Meta's communications head Andy Stone signaled support but did not say whether Meta will honor the signal.
Why it matters
Today a site has to signal its AI training preferences separately to each AI company. A single token in robots.txt, backed by four of the biggest AI developers, would replace that patchwork with one setting. It arrives while publishers are struggling with AI crawlers that feed their content into models, and alongside other defenses such as the RSL standard, which lets publishers charge visiting crawlers, and Cloudflare's default blocking of AI crawlers.
Who it affects
Publishers and website operators are the main audience: they gain one place to state their AI training preferences. The AI companies that run crawlers (OpenAI, Google, Microsoft, Anthropic) take on a commitment to restrict their crawlers when they see the token. Meta is affected too, since it has voiced support without saying it will honor the signal. Microsoft's Bing users and site owners are touched by its promise that sites still appear in Bing search regardless of their preferences.
How to use it
The signal lives in a site's robots.txt file, so publishers would set it there once rather than per company. Microsoft says it will offer granular controls to opt out of AI training or AI grounding, with Bing search visibility unaffected. Google already offers similar controls over AI search features and AI training without hurting search visibility. The source does not give the token's name or syntax.
How solid is it
The core claims are statements by the companies themselves, relayed by The Verge, which cites Search Engine Roundtable as the first to report the news. Anthropic has already adopted the signal, while Google and Microsoft say next year and OpenAI says "coming months," so most of this is intent rather than shipped behavior. The signal is described as proposed within the IETF; the source does not say the IETF has approved it. Meta's comment was added in an October 30th update.
Risks and caveats
Meta has not said whether it will honor the signal, so a major crawler operator may sit outside the arrangement. Commitments from Google, Microsoft and OpenAI are not yet in effect. The source describes no enforcement mechanism or way to verify that companies comply. It also does not say whether the token touches search indexing or archiving crawlers; it concerns AI training preferences.
“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step”
— Andy Stone, Meta communications head, quoted by The Verge