The Crawl Price Index.
Sign in Terminal — €49/mo

The Weekly Crawl

04 Oct 2026

186 domains changed AI-crawler policy across 1057 crawler states, in the Tranco top 50,000.

What changed · the 7 days to 04 Oct 2026
186 domains changed AI-crawler policy across 1,057 individual crawler states, measured on the 30,500 domains that served a readable robots.txt in both editions.
474 of those changes ended in a block and 167 came out of one; the remaining 416 moved between unlisted, allowed and partial without crossing the block line. All 186 movers, with full rows →
Also true this week
Who actually has revenue to protect
9,912 of the 50,000 ranked domains publish an ads.txt, and 12,549 declare schema.org types. These are the sites with something measurable to lose when a crawler reads without sending a reader back. See it →
The observed price
$0.50 / crawl
stackoverflow.com quotes this to ClaudeBot via Cloudflare pay-per-crawl, in x402 format — one of very few published prices anywhere.
Block rates · week on week
Share of the 30,779 domains with a readable robots.txt, inside the Tranco top 50,000.
Every edition since the baseline →
Published 04 Oct 2026. Figures are as published in that edition and are not restated later; corrections are logged on status.
All issues · Method · Get this by email, free