Skip to content
InfoNews

The biggest names in AI are backing a new way to opt out of scraping

Published
Record updated
View JSON

Summary

OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that lets publishers set one AI training preference across several AI companies instead of configuring each one separately. The shared "policy token" signal, proposed by Kevin Kelley of Anthropic within the IETF, would live in a website's robots.txt file, which already indicates which crawlers may scrape a site.