13 Sept 2026

What changed · the 7 days to 13 Sept 2026

201 domains changed AI-crawler policy across 1,067 individual crawler states, measured on the 31,119 domains that served a readable robots.txt in both editions.

484 of those changes ended in a block and 199 came out of one; the remaining 384 moved between unlisted, allowed and partial without crossing the block line. All 201 movers, with full rows →

Also true this week

Sites do not treat all crawlers alike

2,052 domains block a crawler whose stated role is training while blocking no search crawler. 94 do the reverse. That is 21.8 to 1, on the 31,364 domains with a readable robots.txt. The asymmetry is the whole argument for measuring crawlers separately rather than counting “bot blocks”. See it →

The observed price

$0.50 / crawl

stackoverflow.com quotes this to ClaudeBot via Cloudflare pay-per-crawl, in x402 format — one of very few published prices anywhere.

Block rates · week on week

Share of the 31,364 domains with a readable robots.txt, inside the Tranco top 50,000.