The Crawl Price Index.
Sign in Terminal — €49/mo

The Weekly Crawl

27 Sept 2026

156 domains changed AI-crawler policy across 942 crawler states, in the Tranco top 50,000.

What changed · the 7 days to 27 Sept 2026
156 domains changed AI-crawler policy across 942 individual crawler states, measured on the 30,463 domains that served a readable robots.txt in both editions.
414 of those changes ended in a block and 118 came out of one; the remaining 410 moved between unlisted, allowed and partial without crossing the block line. All 156 movers, with full rows →
Also true this week
Machines already pay each other
17,304 endpoints advertise a price in the public x402 registry, median $0.01. Only 18 sit inside the 49,156 ranked domains that capture was joined against. The payment rail exists; it has almost no contact with the web you read. See it →
The observed price
$0.50 / crawl
stackoverflow.com quotes this to ClaudeBot via Cloudflare pay-per-crawl, in x402 format — one of very few published prices anywhere.
Block rates · week on week
Share of the 30,743 domains with a readable robots.txt, inside the Tranco top 50,000.
Every edition since the baseline →
Published 27 Sept 2026. Figures are as published in that edition and are not restated later; corrections are logged on status.
All issues · Method · Get this by email, free