The Crawl Price Index ("the Service") publishes measurements of how the web responds to AI crawlers — block rates, observed refusals and pay-per-crawl prices, payment signals, suffix-group breakdowns, and their movement between editions. Headline figures are published freely, and so is the aggregate series CSV (one row per published edition, aggregates only). The per-domain dataset, its per-edition history, the Watchlist cohort views and their CSV exports, and API access ("the Dataset") are provided to paying subscribers ("Terminal") under the licence below.
Headline figures shown on the public site, and the aggregate series CSV available from the free edition of the dashboard, are licensed CC-BY-4.0: free to quote and republish with attribution to "The Crawl Price Index (crawlpriceindex.com)". We encourage citation.
Subscription to Terminal grants you a single-subscriber, non-exclusive, non-transferable licence to access and use the Dataset for your own internal business or editorial purposes.
You may publish limited quotations and derived insights from the Dataset with attribution (e.g. citing a figure in an article), provided you do not reproduce the Dataset itself or a substantial extract of it.
Your API key is personal to your subscription and rate-limited: 5,000 requests per calendar month per key (raised from 1,000 on 4 September 2026); past that the endpoint answers 429 until the month turns. Write to us before you expect to hit it. Automated bulk mirroring, scraping designed to reconstruct the full Dataset for redistribution, or sharing a key across organisations is a breach of this licence and may trigger rate-limiting, suspension, or termination.
Watchlist lists. Domain lists you save on the Terminal’s Watchlist tab are stored in your own browser only; we do not receive, store or back them up, and clearing site data removes them. The cohort figures and CSV exports the Watchlist produces are part of the Dataset and carry the same licence.
Terminal is billed monthly via Stripe. VAT is handled by our payment provider as merchant of record where applicable. You may cancel at any time; access continues until the end of the paid period and your key is deactivated on lapse.
Subscription pricing may change with at least 30 days’ notice by email before your next renewal. Founding subscribers keep their sign-up rate for at least 24 months from the date they first subscribe; any change after that period applies only from the next renewal following notice. We never change the price of a billing period you have already paid. Terminal is sold monthly; there is no one-off purchase. Snapshot purchases made before 3 September 2026 remain valid for their 48-hour download window.
Measurements reflect what public robots.txt files and crawler-facing HTTP responses disclosed at scan time. robots.txt figures record stated preferences, which sites may or may not enforce. Observed prices change without notice. The Service is provided "as is", without warranty of accuracy, completeness, or fitness for a particular purpose. It is not legal, financial, or licensing advice.
To the maximum extent permitted by law, our liability arising from your use of the Service is limited to the amount you paid in the preceding three months.
We may update these terms; material changes will be dated here. 4 September 2026: API cap raised to 5,000 requests a month; Watchlist (browser-local lists) and the aggregate series CSV added. Questions, licensing enquiries, or bulk/enterprise access: hello@crawlpriceindex.com.