The Crawl Price Index.
Sign in Terminal — €49/mo

The Weekly Crawl

20 Sept 2026

1349 domains changed AI-crawler policy across 9706 crawler states, in the Tranco top 50,000.

What changed · the 14 days to 20 Sept 2026
1,349 domains changed AI-crawler policy across 9,706 individual crawler states, measured on the 30,395 domains that served a readable robots.txt in both editions.
1,064 of those changes ended in a block and 7,864 came out of one; the remaining 778 moved between unlisted, allowed and partial without crossing the block line. All 1,349 movers, with full rows →
Also true this week
A robots.txt is a statement, not a door
4,988 of the 50,000 ranked domains serve a readable robots.txt and then refuse an identified crawler that obeys it. Declared policy and enforced policy are two different measurements, and only one of them is in the file. See it →
The observed price
$0.50 / crawl
stackoverflow.com quotes this to ClaudeBot via Cloudflare pay-per-crawl, in x402 format — one of very few published prices anywhere.
Block rates · week on week
Share of the 30,742 domains with a readable robots.txt, inside the Tranco top 50,000.
Every edition since the baseline →
Published 20 Sept 2026. Figures are as published in that edition and are not restated later; corrections are logged on status.
All issues · Method · Get this by email, free