Enter a domain. We show what it declares to 18 tracked AI crawlers in its robots.txt, and how that compares with the domains in this edition’s scanned frame. This is a measurement of declared policy — not a valuation, a recommendation, or a forecast. No signup needed.
checking index…
Step 1 · your domain
Your AI-crawler policy fingerprint ·
—
of 18 tracked crawlers blocked in robots.txt
—
of domains with a readable robots.txt block fewer crawlers than you
—
rank in the CPI-50K v1 frame
Percentile among the domains with a readable robots.txt, by how many of the 18 tracked crawlers each one blocks.
Get told when this changes
AI-crawler policy is not static — domains flip between blocked and allowed. We scan the whole top 50,000 twice a week. If anything changes for your domain, you get one email naming exactly what changed. Nothing else.
Step 2 · publish your own terms
If you're on Cloudflare: enable AI Crawl Control, join the Pay Per Crawl beta, and set your price — minimum $0.01, one price for all charged crawlers. Cloudflare collects and pays out; you never touch the plumbing.
If you're a publisher with real bot traffic: TollBit runs a bot paywall on ~7,000 sites — you keep 100% of your posted rate; AI buyers pay TollBit's fee on top.
Everyone: declare machine-readable terms with RSL and robots.txt so your policy is visible to every compliant crawler. Paste the two files below — generated for your domain at whatever price you choose. We do not suggest a number: we measure what sites declare, we do not advise what they should charge.
$✓ UpdatedYour choice — we do not recommend a figure. It is written into rsl.xml, and noted as a comment in robots.txt.
robots.txt — add these lines
rsl.xml — save at your site root, licence terms machines can read
Get the movers by email
Your fingerprint and the benchmarks above move as the web reprices itself. The Weekly Crawl delivers the movers — which domains changed policy, block-rate shifts, observed prices.
Want every domain’s row, suffix-group detail and per-edition history? Terminal — €49/mo.