Explore data
The free edition of the dashboard carries this edition’s real figures on two tabs — the Briefing (citable sentences, each with its claims-sheet ID) and Full detail (every headline measure with its denominator). The other eight tabs — Crawlers, Policy layer, Policy changes, Domains, Segments, Watchlist, Wire evidence and Machine payments — and the per-domain downloads are Terminal. It runs on live data, so it is shown in the dashboard itself rather than copied here.
Below: this edition’s headline cuts, refreshed with every census.
The cuts below are the current edition, in full and free to cite with attribution. They are the same figures behind the dashboard above — what a subscription adds is the per-domain layer underneath them, not these.
One edition-over-edition comparison, not a trend. Domains entering or leaving the frame are excluded, so frame churn is never counted as a policy change.
A popularity ranking lists domains, not working websites. We census the frame itself so every rate we publish has an honest denominator — and we keep declared policy and enforced access as separate measurements.
Share of domains with a readable robots.txt that disallow each crawler. Training bots are walled harder than the search crawlers that send traffic back.
Group the frame by domain ending and the block rates run from one extreme to the other. That spread is real and it is worth knowing — but a suffix is a registration choice, not a country, an owner, an audience or a place where a server sits.
Groups also differ in rank composition, so part of any gap between them is rank rather than policy. All suffix groups, and why this is not a map →
Each edition we record the declared block rate for every tracked crawler and archive it. The series is short, so this chart plots one dot per published edition and does not join them, and we do not describe it as a trend anywhere on this site. Prices, 402 responses and enforcement behaviour are observed live by an identified crawler and are not recoverable from web archives — unlike robots.txt, which the Wayback Machine preserves.
When a request presenting a named AI crawler knocks, the answer is one of three things — measured on the wire this edition, beside what our own self-identified crawler (the honest-identity control) got from the same door.
Alongside the census we capture a public registry of endpoints advertising a price a machine can pay directly, settled in stablecoin. It is a real, working market — and it is almost entirely developer APIs and tools rather than the sites people visit.
Advertised, opt-in acceptance in a public registry — never transactions, volume or revenue. This is a stablecoin rail, distinct from Cloudflare pay-per-crawl, though both return HTTP 402.