Declared AI-crawler blocking varies widely across domain suffixes, and it is tempting to read that as a map of national attitudes. It is not one: a suffix is a string at the end of a name, and the three reasons a group’s rate moves — set out below, before the numbers — are none of them national.
The dashboard’s Segments tab compares each group against the whole-index rate and against rank bands, which is the cut that begins to separate these effects. This page shows the raw distribution only.
—
—
Share of each group’s domains that explicitly block at least one of the 18 tracked AI crawlers, among domains serving a readable robots.txt. Sample size (n) is shown for every group, and a rate is published only above the floor named above — the same floor the Terminal’s Segments tab uses, so the two pages name the same highest and lowest groups.
.uk differs between pagesThis page keys a domain by its country-code suffix and lists a two-label public suffix as its own group (.co.uk is a group here, separate from .uk). The Terminal’s Segments tab and the band table on Explore key by the last label only, so a .co.uk domain appears under .uk there. The two pages’ n for .uk therefore differ, and neither is wrong: they are two conventions, stated here.
Highest observed: —. Lowest: —. Neither is a ranking of anything but this measurement. Groups below the floor are listed with their n and no rate: —.
The headline rate hides which crawler is being blocked. Some groups show a wider gap between training-role and search-role crawlers than others. The same three confounds above apply to every column.
| ccTLD | GPTBot | ClaudeBot | Google-Ext. | Any AI |
|---|