The open internet is quietly installing a toll booth for machines. We measure the tolls — observed prices, paywalls, and block rates across the web — and publish them every week.
When an AI crawler requests a page, a growing number of sites answer with HTTP 402 Payment Required instead of the content. A few attach an actual number. In this scan, Stack Overflow quoted USD 0.50 per crawl to Anthropic's ClaudeBot through Cloudflare's pay-per-crawl, returned in the same x402 settlement format agents use to pay autonomously. That single quote is the leading edge of a market that barely has public prices. We exist to collect them.
Across the top 2,000 sites on the web, fewer than one in five block any given AI crawler in robots.txt. Among news and reference publishers, the rate is roughly four to five times higher — the sites whose text models most want are the sites fighting hardest to gate it. And publishers now discriminate between bots: training crawlers get blocked far more than the search bots that still send readers back.
These are the whole-web figures, refreshed weekly. The full per-domain table, publisher panel, and country cuts are in the data feed.
| AI crawler | Blocked in robots.txt | Rate |
|---|
No standard has won, so the crawl economy speaks in dialects. We fingerprint each one across the web and track its spread. Every dialect below was observed in the current scan.
→ New: World editions — AI-crawler blocking by country
Headline numbers are free and meant to be cited. Depth — full per-domain history, the publisher panel, per-country and per-language editions, and the machine-readable feed — is for subscribers. Fittingly, our own data is available to AI agents through pay-per-crawl: the index charges for crawls, in an index of what crawls cost.