AI crawlers

Eight crawlers decide whether your writing reaches a generative answer. Domduck reads every domain’s robots.txt on a schedule and records what it said, so the question “when did this site start blocking GPTBot” has an answer rather than a guess.

Last full reading 12 October 2026, across 58 domains.

What changed todayThe quarterly index
Sites that block every crawlerread 2026-10-12
1.7%carry Disallow: /
for every crawler
2026-08-021.1% – 5.2%2026-10-12
A catch-all Disallow: / on 1 of 58 domains read. It blocks all eight agents at once, whether or not the file names them.
Domains read58on the most recent pass
Catch-all block1.7%Disallow: / for everyone
Serve llms.txt18.97%the proposed standard
Agents tracked8three states each
Changes, 30 days24+policy flips observed

How the web answers, agent by agent

read 2026-10-12
GPTBotOpenAI5.2%

OpenAI’s crawler for gathering web content used to train its models.

ClaudeBotAnthropic5.2%

Anthropic’s crawler, used to gather web content for Claude.

PerplexityBotPerplexity5.2%

Perplexity’s crawler, which indexes pages so they can be cited in its answers.

Google-ExtendedGoogle5.2%

Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.

CCBotCommon Crawl5.2%

The crawler behind Common Crawl, a free public archive of the web that most open training corpora are built from.

BytespiderByteDance5.2%

ByteDance’s crawler, gathering content used for TikTok’s search and for Doubao.

Applebot-ExtendedApple3.4%

The same shape as Google-Extended: a control token, not a crawler. Applebot fetches the page either way; this decides whether the content may train Apple Intelligence.

OAI-SearchBotOpenAI1.7%

The crawler behind ChatGPT’s search surface: it fetches pages so they can be cited in answers.

BlockedAllowed by nameNot named

“Not named” is a third state and not a rounding of “allowed”. A file that never mentions an agent leaves it exposed to a catch-all rule added later, so the two answers have different futures. The blocked share counts the verdict actually in force, including agents caught by a catch-all they are not named in.

What changed lately

last 30 days
  1. 12 October 2026 (2026-10-12)
  2. 11 October 2026 (2026-10-11)
  3. 10 October 2026 (2026-10-10)