AI companies back a shared robots.txt opt-out signal

OpenAI, Google, Microsoft and Anthropic are backing a shared signal in the robots.txt file that lets website owners say whether AI crawlers may use their content to train models. TechRadar reports that OpenAI will honor the signal and quoted the company encouraging other AI providers to do the same.

The robots.txt file sits in a website’s root directory and gives bots guidance about how to access the site. The article says the signal can block many bots that follow those rules, but it will not stop scrapers that ignore them.

TechRadar also describes other ways to restrict bot traffic: require visitors to create accounts, limit requests from individual IP addresses, or use CAPTCHAs. The article notes trade-offs: account requirements can make information harder for people to access, some scrapers use rotating IP addresses, and CAPTCHAs can frustrate users or be solved by some AI models.

The report presents these measures as imperfect options. It says restrictions intended to limit scraping can also affect the experience of human visitors.