The Crawl Price Index.
Sign in Terminal — €49/mo

The Weekly Crawl

06 Sept 2026

183 domains changed AI-crawler policy across 1207 crawler states, in the Tranco top 50,000.

What changed · the 7 days to 06 Sept 2026
183 domains changed AI-crawler policy across 1,207 individual crawler states, measured on the 31,200 domains that served a readable robots.txt in both editions.
419 of those changes ended in a block and 282 came out of one; the remaining 506 moved between unlisted, allowed and partial without crossing the block line. All 183 movers, with full rows →
Also true this week
Domain suffix is not a country
Grouped by suffix, .co.uk blocks at least one AI crawler on 39% of its 403 domains; .cn on 7.1% of 239. A suffix indicates neither operator location nor audience, and these are small groups. It is a cut of the data, not a map. See it →
The observed price
$0.50 / crawl
stackoverflow.com quotes this to ClaudeBot via Cloudflare pay-per-crawl, in x402 format — one of very few published prices anywhere.
Block rates · week on week
Share of the 31,441 domains with a readable robots.txt, inside the Tranco top 50,000.
Every edition since the baseline →
Published 06 Sept 2026. Figures are as published in that edition and are not restated later; corrections are logged on status.
All issues · Method · Get this by email, free