The Crawl Price Index.

The Weekly Crawl

30 Aug 2026

179 domains changed AI-crawler policy across 1136 crawler states, in the Tranco top 50,000.

Corrections (2) — expand
Note, 4 September 2026. This issue’s change window ran from the 24 August validation scan, which is not a published edition, to the 30 August edition; its counts are therefore not comparable with the edition-to-edition change sets on the dashboard. Its denominator is rounded (31,500; the edition parsed 31,488). Figures are left as emailed; see status for the record.
Further note, 4 September 2026. Four more things in this issue are corrected here rather than rewritten. (1) “Both editions” in the change sentence and “week on week” in the block-rate heading describe a five-day window from a scan that is not an edition to one that is; the block-rate movements in the table are over that same five-day window, not a week. (2) The change arithmetic: 527 more restrictive plus 266 less restrictive is 793 of the 1,136 crawler states; the remaining 343 were reclassified between allowed, partial and unlisted — neither into nor out of a block — and the issue did not say so. (3) The $0.50 price was recorded by the 331-domain identity-matrix panel, not by the wide probe whose “named a price” count appears on the home page; the two instruments are counted separately and never summed. (4) The price was a 402 response carrying a crawler-price header, which is the Cloudflare pay-per-crawl form; it is not an x402 offer — x402 is a different rail that also answers 402, and the sentence below names both as if they were one. The emailed text stands unaltered below.
What changed · the 5 days to 30 Aug 2026
179 domains changed AI-crawler policy across 1136 individual crawler states, measured on the 31,173 domains that served a readable robots.txt in both editions.
527 changes made a crawler's state more restrictive, 266 less. All 179 movers, with full rows →
Also true this week
The opt-out standards are still small
TDM Reservation Protocol was found on 3.95% of 2,000 probed domains. Content-Signal, RSL and TDM together are declared by a low-thousands minority of the 31,500 domains with a readable robots.txt. See it →
The observed price
$0.50 / crawl
stackoverflow.com quotes this to ClaudeBot via Cloudflare pay-per-crawl, in x402 format — one of very few published prices anywhere.
Block rates · week on week
Share of the 31,500 domains with a readable robots.txt, inside the Tranco top 50,000.
Every edition since the baseline →
Published 30 Aug 2026. Figures are as published in that edition and are not restated later; corrections are logged on status.
All issues · Method · Get this by email, free