Coverage · edition of 2026-10-04

Anatomy of the frame

Every figure this index publishes is a share of the 30,779 domains that served a readable robots.txt in the 50,000-domain frame — not of 50,000, and not of the web. This page is where that number comes from, and what is inside it.

Frame

50,000

ranked domains

Readable robots.txt

30,779

61.56% — the denominator

No reading

19,221

38.44% — broken out below

Block ≥1 crawler

5,031

16.35% of the denominator

Widths are proportional to domain counts. The domains that answered a request and still returned no robots.txt outnumber those that never returned a usable response — and every branch below is measured by the scanner, not inferred.

Why 19,221 domains told us nothing

The scanner records a reason for every domain it cannot read. These 9 groups cover 75 distinct raw reasons and sum to 19,221 exactly, with nothing left over.

Reason	Domains	Of the 19,221
Name never resolved, or the connection was refused	4,177	21.73%
Refused an identified crawler (403 or 429)	4,092	21.29%
No robots.txt file (404 or 410)	3,764	19.58%
Answered 200, but the body was not a robots.txt	2,503	13.02%
TLS would not negotiate	1,754	9.13%
Connected, then timed out	1,733	9.02%
Connection dropped mid-transfer	523	2.72%
Other HTTP status	379	1.97%
Server error (5xx)	296	1.54%

10,359 of them answered the door and still said nothing — 53.89% of the group, more than everything that failed to connect (8,187), with 675 more that connected and answered a server error. A 404 is not the same as an unlisted crawler: 3,764 domains have no robots.txt at all and have therefore made no declaration about anything, while an unlisted crawler means a file exists and does not mention it.

Inside the 30,779

These branches are computed from the census itself. Each level sums to the one above it.

Grouping: Cloudflare’s — training and user-initiated crawlers together as machine consumption, against search. The home page’s asymmetry card uses the index’s own role tags (training against search, user-initiated on neither side), which is why it reads 2,167 / 96. Four in five domains that publish a robots.txt block none of the 18 tracked crawlers. Of those that do block, most draw no distinction between a crawler that sends readers back and one that does not.

Branch	Domains	Of 30,779
Block none of the 18	25,748	83.65%
Name none of the 18 at all	23,971	77.88%
Named one and explicitly allowed it	928	3.02%
Path-scoped rule only, never the whole site	849	2.76%
Block at least one	5,031	16.35%
Search and machine-consumption alike	2,756	8.95%
Machine-consumption only, search left open	2,180	7.08%
Search only, machine-consumption left open	95	0.31%

The 1,026 that block exactly one crawler

Blocking a single named crawler and nothing else cannot be a copied snippet or a hosting default. Someone chose that one.

Crawler	Sole target	Share
GPTBot	285	27.78%
CCBot	223	21.73%
Bytespider	217	21.15%
Amazonbot	91	8.87%
meta-externalagent	80	7.80%
ClaudeBot	34	3.31%
Google-Extended	31	3.02%
Applebot-Extended	27	2.63%
7 other crawlers	38	3.70%

How many crawlers each blocker blocks

Crawlers blocked	Domains	Of blockers
1	1,026	20.39%
2–7	1,569	31.19%
8 — the copied list and its variants	204	4.05%
9–17	1,995	39.65%
All 18	237	4.71%

Only 237 domains block every tracked crawler without exception — 0.77% of everything with a readable robots.txt. Blanket refusal, as a stated policy, is rare.

Wildcard policy coverage

Raw-body wildcard analysis is unavailable for this edition in this view. This is a coverage gap, not a finding of zero wildcard restrictions. The named-crawler figures above come from the parsed census and remain available.

At the level of individual states

30,779 domains × 18 crawlers is 554,022 individual declarations.

State	Cells	Of 554,022	Meaning
unlisted	501,073	90.44%	The file exists and does not mention this crawler
blocked	38,536	6.96%	Named, with `Disallow: /`
allowed	7,595	1.37%	Named, and explicitly permitted
partial	6,818	1.23%	Named, with a path-scoped disallow only

Nine declarations in ten are silence. Most sites have said nothing about most AI crawlers, and that is the most honest one-line summary of the declared-policy layer.

By rank band

CPI-50K v1 rank band	Parsed	Block ≥1
1-100	66	27.27%
101-500	271	30.63%
501-1000	350	24.86%
1001-5000	2,747	20.93%
5001-10000	3,194	18.07%
10001-25000	9,222	16.1%
25001-50000	14,929	14.78%

Blocking declines with rank below the top thousand. This gradient is what reconciles our headline with indices that sample deeper: extend this frame to 100,000 along the observed curve and it predicts a materially lower rate.

How to read this page

The census figures above are computed from the edition of 2026-10-04, from the census file each time this page is built. Raw-body wildcard analysis is unavailable in this view for this edition. These figures describe declared robots.txt policy, not observed crawler access.

The per-domain rows behind these counts are the subscription. The shape is free, and free to cite with attribution.