
Teardown & Case Study10 min read
AI Crawlers in robots.txt: What 8 Major Sites Block and Allow
We read the robots.txt files of NYT, Forbes, the Guardian, Wikipedia and four more. The smart ones block AI training crawlers and keep AI search crawlers open.
