Google-Extended

Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.

Measured across 58 domains read on 12 October 2026, against 14.7% on 2 August 2026.

Check one domainAll eight agents
How the corpus answers Google-Extendedread 2026-10-12
5.2%of sites
disallow it
Blocked 5.2%Allowed by name 6.9%Not named 89.7%
2026-08-023.2% – 18.6%2026-10-12
Block rate over 72 recorded days. The denominator is stored with each point, so the line is not restated when the corpus grows.
Operated byGoogle
Blocked by5.2%3 domains
Allowed by name6.9%the file names it
Not named89.7%the file is silent
Sample58domains read
robots.txt tokenGoogle-Extendedcase-insensitive

What blocking Google-Extended costs

Your content stops being used to train or ground Google’s generative products.

What it does not cost

IT DOES NOT AFFECT SEARCH RANKING. Google states this explicitly, and it is the single most misunderstood entry in this table. Blocking Google-Extended is the cheapest of the eight decisions and many sites avoid it believing the opposite.

Because it is a token rather than an agent, you will never see a request from it in your logs. There is nothing to verify and nothing to rate-limit.

Google’s own documentation →

How to block itRFC 9309
User-agent: Google-Extended
Disallow: /
User-agent: Google-Extended
Disallow:

Allowed, and said out loud.

To allow it and say so, use a bare Disallow: with nothing after it. A file that never names an agent leaves it in a third state, and a catch-all added later blocks it without anyone deciding to.

Sites that changed their mind

last 90 days
  1. 7 October 2026 (2026-10-07)
    • ap.orgGoogle-Extended was blocked between 15 September 2026 and 7 October 2026.
  2. 5 October 2026 (2026-10-05)
    • thelancet.comGoogle-Extended was unblocked between 15 September 2026 and 5 October 2026.
  3. 3 October 2026 (2026-10-03)
    • kniga-online.comGoogle-Extended was unblocked between 15 September 2026 and 3 October 2026.
    • claude.aiGoogle-Extended was unblocked on 3 October 2026.
    • aflamk1.netGoogle-Extended was unblocked between 15 September 2026 and 3 October 2026.
  4. 2 October 2026 (2026-10-02)
    • dorama.landGoogle-Extended was unblocked between 15 September 2026 and 2 October 2026.
    • claude.aiGoogle-Extended was blocked on 2 October 2026.
  5. 1 October 2026 (2026-10-01)
    • depositphotos.comGoogle-Extended was blocked between 17 September 2026 and 1 October 2026.
  6. 30 September 2026 (2026-09-30)
    • gemidos.tvGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • getmodsapk.comGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • apuestasroyal.comGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • porn-comic.comGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • bigbadtoystore.comGoogle-Extended was unblocked between 21 August 2026 and 30 September 2026.
    • vid30s.comGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • distintas.netGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • fullhdfilmizlesene.lifeGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • romeo.comGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • insidehighered.comGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
    • livecamrips.toGoogle-Extended was unblocked between 15 September 2026 and 30 September 2026.
  7. 29 September 2026 (2026-09-29)
    • emojicombos.comGoogle-Extended was unblocked between 15 September 2026 and 29 September 2026.

The other seven

GPTBotOpenAI

OpenAI’s crawler for gathering web content used to train its models.

OAI-SearchBotOpenAI

The crawler behind ChatGPT’s search surface: it fetches pages so they can be cited in answers.

ClaudeBotAnthropic

Anthropic’s crawler, used to gather web content for Claude.

PerplexityBotPerplexity

Perplexity’s crawler, which indexes pages so they can be cited in its answers.

Applebot-ExtendedApple

The same shape as Google-Extended: a control token, not a crawler. Applebot fetches the page either way; this decides whether the content may train Apple Intelligence.

CCBotCommon Crawl

The crawler behind Common Crawl, a free public archive of the web that most open training corpora are built from.

BytespiderByteDance

ByteDance’s crawler, gathering content used for TikTok’s search and for Doubao.

How this page is made

The explanation above is written by a person. The numbers are computed from Domduck’s own daily reads of each domain’s robots.txt, parsed to RFC 9309 rather than searched for a string: a rule binds to the nearest preceding group of user-agent lines, the longest matching pattern wins, and Allow breaks a tie. No prefix matching is done, so a group naming Google is not treated as a policy about Google-Extended. What the collectors have read.