Training crawlers vs retrieval crawlers
A training crawler collects pages to build a model. A retrieval crawler fetches a page so an answer can cite it. Blocking one does nothing to the other.
A training crawler collects pages to build a model. A retrieval crawler fetches a page so an answer can cite it. They are separate agents with separate names, and blocking one does nothing to the other.
Most published advice treats the eight AI agents as one decision. They are eight decisions with different costs, and the split below is the one that decides whether blocking costs you readers.
| Purpose | Agents Domduck tracks | Blocking it costs you |
|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Bytespider | Future inclusion in training data. Nothing already collected comes back out. |
| Retrieval and citation | OAI-SearchBot, PerplexityBot | Citations in answers, which is a link with a reader attached |
| Control token, not a crawler | Google-Extended, Applebot-Extended | Use of your pages in that company's AI features. Not the crawl and not the ranking. |
The one people get wrong
Google-Extended is not a crawler. Googlebot still fetches the page, and the token decides only whether the content feeds Google's generative features. Blocking it does not affect Search ranking at all, and the belief that it does is the most common mistake in this subject.
The same shape applies to Applebot-Extended: Applebot still crawls for Siri and Spotlight, and the Extended token governs training use only.
What Domduck records
Each agent carries its own verdict per day, so the training and retrieval halves can be read separately. The numeric series is the count of blocked agents out of eight, which is the number that goes on a chart, and the per-agent detail sits under it.
On 2026-08-03, across 2,228 domains, the training crawlers were blocked far more often than the retrieval ones: GPTBot at 17.0% against OAI-SearchBot at 6.7%. Sites that thought about it mostly decided to be quoted and not to be trained on.
Why it matters
If you block a training crawler expecting to disappear from an AI answer, you have paid a cost and bought nothing. The retrieval crawler is a different name and it is still allowed.
The reverse mistake is worse: a catch-all Disallow: / written to stop scraping takes the citation crawlers with it, and citations were the part sending you readers.
Related
- robots.txtrobots.txt is a file at the root of a site that tells crawlers which paths they may fetch, one group of rules per user agent.
- Allowed, disallowed, unspecifiedUnspecified is the third answer a robots.txt file can give about a crawler: the file exists, and it never mentions that agent at all.
Primary source: OpenAI's crawler documentation