The Weekly Crawl
179 domains changed AI-crawler policy across 1136 crawler states, in the Tranco top 50,000.
crawler-price header, which is the Cloudflare pay-per-crawl form; it is not an x402 offer — x402 is a different rail that also answers 402, and the sentence below names both as if they were one. The emailed text stands unaltered below.fastcompany.com #470 identical edit on fastcoexist.com ClaudeBot, anthropic-ai, PerplexityBot and 5 others unlisted → blocked; OAI-SearchBot, ChatGPT-User, Perplexity-User unlisted → partial; Google-Extended allowed → partial; CCBot allowed → blocked; meta-externalagent blocked → partial |
inc.com #593 ClaudeBot, anthropic-ai, PerplexityBot and 5 others unlisted → blocked; OAI-SearchBot, ChatGPT-User, Perplexity-User unlisted → partial; CCBot partial → blocked; meta-externalagent blocked → partial |
challenges.fr #7788 anthropic-ai, PerplexityBot, Bytespider and 5 others unlisted → blocked; OAI-SearchBot, Perplexity-User unlisted → partial; ChatGPT-User, Claude-Web blocked → partial |
pourquoidocteur.fr #48247 GPTBot, ClaudeBot, anthropic-ai and 9 others unlisted → blocked; OAI-SearchBot, ChatGPT-User, Claude-Web and 2 others unlisted → allowed |
businessweekly.com.tw #22581 GPTBot, ClaudeBot, Google-Extended and 5 others unlisted → blocked; OAI-SearchBot, PerplexityBot, Perplexity-User unlisted → allowed |
gnome-look.org #29805 identical edit on pling.com OAI-SearchBot, ChatGPT-User, Perplexity-User and 5 others unlisted → blocked; CCBot blocked → unlisted |
| GPTBot | 15.9% | ▲ +0.1 pts | |
| CCBot | 15.5% | ▲ +0.2 pts | |
| Bytespider | 14.1% | ▲ +0.2 pts | |
| ClaudeBot | 13.4% | ▲ +0.1 pts | |
| Google-Extended | 12.4% | ▲ +0.1 pts | |
| meta-externalagent | 11.7% | ▲ +0.1 pts |