OpenAI, Google, Microsoft, and Anthropic back shared AI scraping opt-out

OpenAI, Google, Microsoft, and Anthropic say they will adopt a shared standard that lets publishers signal whether they want their sites used for AI training. The proposal, called a “policy token,” would let publishers set preferences across multiple AI companies instead of contacting each one separately.

Proposed by Anthropic technical lead Kevin Kelley through the Internet Engineering Task Force (IETF), the signal would appear in a website’s robots.txt file. That file already tells web crawlers which parts of a site they may access for purposes such as search indexing, archiving, and AI training.

The companies say their AI crawlers will respect the shared token. Anthropic has already adopted it; Google and Microsoft plan to support it next year, while OpenAI says it will implement the standard in the coming months.

Microsoft plans to offer more specific controls, allowing publishers to opt out of AI training, AI grounding, or both. Grounding means directing an AI model to verified information to reduce hallucinations. Microsoft says sites will still appear in Bing search results regardless of their preferences.

The proposal comes as publishers face continued scraping by AI crawlers. Other responses include the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers, and Cloudflare’s default blocking of AI crawlers.