Methodology — The Crawl Price Index

The short version

Once a week we read the robots.txt and homepage response headers of the Tranco top 50,000 domains using our own identified, cryptographically signed crawler. We record what each site tells AI crawlers: who is blocked, who is allowed, whether a price or payment wall is quoted, and what changed since last week. We publish the aggregates free and the per-domain detail to subscribers.

The distinction that matters most. “GPTBot is disallowed in this site’s robots.txt” is an observation. “15.3% of the scanned web blocks GPTBot” is a derivation from those observations. “A site of this size could charge $0.05 per crawl” is an inference — our judgement, not a measurement. We label all three differently everywhere they appear, and we never present the third as the first.

What we measure

Signal	How	Type
Robots stance per crawler, per domain	GET `/robots.txt` with our identified user agent; parse User-agent / Disallow / Allow groups. Values: blocked, allowed, partial, unlisted, no_robots.	observed
Observed price	Recorded only when a site returns a price in a machine-readable payment response — HTTP 402 with a `crawler-price` or payment header, or an equivalent x402 offer. We publish the quote and the domain that gave it.	observed
Payment signals	Status codes and headers on identified-crawler requests: 402 responses, TollBit-style token walls, licensing redirects, `payment:free` declarations.	observed
Enforcement	Compares a site’s declaration against what an identified crawler actually receives. Publisher panel only (~49 domains), so directional, not projectable.	observed (panel)
Block rates	Share of parsed domains blocking each crawler. Denominator is domains successfully parsed, never the nominal top-N.	derived
Country editions	Segmented by country-code TLD, published where at least 8 domains are present. ccTLD is a proxy for country, not for publisher nationality or audience.	derived
Trends & movers	Difference between consecutive weekly snapshots in our archive.	derived
Suggested price band checker only	Our judgement, anchored to observed quotes and the domain’s rank. Labelled as judgement wherever it appears.	inferred

What we do not measure

Revenue actually earned by any site.
Traffic volumes of any site. The revenue estimates in the checker come from figures you enter.
Prices for domains that do not publish one — we never model or extrapolate a price.
The value of private AI licensing deals.
Whether an AI company honoured a payment request after it was made.

How we identify ourselves

Our crawler sends a truthful user agent and signs every request cryptographically, so any site can verify that traffic claiming to be ours really is:

User-Agent: CrawlPriceIndexBot/1.0 (crawl-economy measurement;
            robots.txt study; contact: hello@crawlpriceindex.com)
Signature-Agent: "https://crawlpriceindex.com"
Signature-Input: sig=("@authority" "signature-agent");created=…;keyid="…";tag="web-bot-auth"

Signatures follow RFC 9421 under the Web Bot Auth profile. Our public key directory is served at /.well-known/http-message-signatures-directory. We have applied to Cloudflare’s verified bots programme using signature verification.

Crawl etiquette

We request robots.txt and the homepage only. We do not crawl article pages and we store no page content.
Requests are rate-limited and back off automatically when a host signals pressure (429/403 rates above threshold reduce concurrency and increase spacing).
A sweep is spread over days rather than hammering hosts in a burst.
Spoofed crawler identities are used only on a small, documented publisher panel to test enforcement — never at scale, and never to obtain content.
To be excluded from the index entirely, email hello@crawlpriceindex.com from a domain-verifiable address.

Coverage and freshness

Frame: the Tranco top 50,000 (a research ranking that aggregates several sources and resists manipulation), plus a fixed publisher panel of ~49 news and reference sites.
Parsed: not every domain answers. We publish the parsed count alongside the frame every week and always compute rates against the parsed count.
Cadence: weekly full sweeps, run automatically.
History: begins the week we first observed a signal. It cannot be backfilled — by us or by anyone.

Known limitations

Declaration is not enforcement. Our headline block rates measure what sites say. Our enforcement metric measures a small panel of what sites do, and the two differ substantially.
robots.txt is not the whole story. Sites also gate crawlers at the edge, by ASN, by fingerprint or by CAPTCHA. We do not see those from a single identified request.
Prices are sparse. Very few sites currently quote a machine-readable price. That sparsity is itself the finding, and we report the count rather than smoothing it.
ccTLD ≠ country. A .io domain is rarely British Indian Ocean Territory.
Panel size. Enforcement and payment-signal findings come from ~49 domains. We report n every time.
One vantage point. We crawl from one network. Sites that vary responses by geography or ASN may present differently elsewhere.

Corrections

If we get something wrong, we fix it in the next weekly edition and record it in the machine-readable corrections array with the date, what changed and why. We do not silently amend past editions. Report an error to hello@crawlpriceindex.com and it will be acknowledged in the next edition.

Reproduce our work

Every figure we publish is regenerated weekly from a full sweep. To check any headline number yourself:

# the aggregates behind the homepage
curl -s https://crawlpriceindex.com/index.json

# how we measure, machine-readable
curl -s https://api.crawlpriceindex.com/v1/methodology

# any domain's stance, free
curl -s "https://api.crawlpriceindex.com/v1/check?domain=example.com"

# and the site's own robots.txt, to check us against the source
curl -s https://example.com/robots.txt

If our stance for a domain disagrees with that site’s live robots.txt, that is a bug and we want to hear about it. Subscribers get the full per-domain dataset as JSON or CSV, with a per-customer watermark.

Independence

The Crawl Price Index is funded by subscriptions. We take no payment from AI companies, publishers, CDNs or licensing platforms in exchange for coverage, ranking or inclusion. We link to Cloudflare, TollBit and RSL as setup routes for site owners with no commercial relationship to any of them. Our own site publishes machine-readable licensing terms and charges AI training crawlers, at a price derived from the index itself — disclosed in rsl.xml.