OpenAI Solves the Scraping Problem It Created, Eventually

The claim: OpenAI states it will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers.

The reality: Publishers still lack granular control over their intellectual property and must trust a scraper that has historically ignored standard robots.txt exclusions until threatened with lawsuits.

Why should I care? Could be big
Trusting an AI lab to honor robots.txt without independent verification is a bold compliance strategy.

Read the original: Making it easier for websites to express AI training preferences