AI crawlers
Eight crawlers decide whether your writing reaches a generative answer. Domduck reads every domain’s robots.txt on a schedule and records what it said, so the question “when did this site start blocking GPTBot” has an answer rather than a guess.
Last full reading , across 2,228 domains.
How the web answers
read 2026-08-03| Crawler | Blocked | Allowed by name | Not named |
|---|---|---|---|
| GPTBot OpenAI | 17.0% | 5.4% | 80.4% |
| OAI-SearchBot OpenAI | 6.7% | 5.0% | 91.2% |
| ClaudeBot Anthropic | 16.1% | 4.0% | 83.1% |
| PerplexityBot Perplexity | 10.6% | 4.8% | 87.4% |
| Google-Extended Google | 14.5% | 4.3% | 84.2% |
| Applebot-Extended Apple | 13.6% | 1.6% | 88.2% |
| CCBot Common Crawl | 17.5% | 2.6% | 83.3% |
| Bytespider ByteDance | 16.3% | 1.8% | 85.5% |
“Not named” is a third state and not a rounding of “allowed”. A file that never mentions an agent leaves it exposed to a catch-all rule added later, so the two answers have different futures. The blocked column counts the verdict actually in force, including agents caught by a catch-all they are not named in.
What each one is for
- GPTBot
OpenAI’s crawler for gathering web content used to train its models.
- OAI-SearchBot
The crawler behind ChatGPT’s search surface: it fetches pages so they can be cited in answers.
- ClaudeBot
Anthropic’s crawler, used to gather web content for Claude.
- PerplexityBot
Perplexity’s crawler, which indexes pages so they can be cited in its answers.
- Google-Extended
Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.
- Applebot-Extended
The same shape as Google-Extended: a control token, not a crawler. Applebot fetches the page either way; this decides whether the content may train Apple Intelligence.
- CCBot
The crawler behind Common Crawl, a free public archive of the web that most open training corpora are built from.
- Bytespider
ByteDance’s crawler, gathering content used for TikTok’s search and for Doubao.