Google-Extended
Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.
14.5% of the 2,228 domains Domduck read on disallow it, against 14.7% on .
What blocking Google-Extended costs
Your content stops being used to train or ground Google’s generative products.
What it does not cost
IT DOES NOT AFFECT SEARCH RANKING. Google states this explicitly, and it is the single most misunderstood entry in this table. Blocking Google-Extended is the cheapest of the eight decisions and many sites avoid it believing the opposite.
Because it is a token rather than an agent, you will never see a request from it in your logs. There is nothing to verify and nothing to rate-limit.
How to block it
or allow it explicitlyUser-agent: Google-Extended
Disallow: /To allow it and say so, use a bare Disallow: with nothing after it, which RFC 9309 defines as “nothing is disallowed”. That matters more than it looks: a file that never names an agent leaves it in a third state, neither allowed nor blocked, and a catch-all rule added later will block it without anyone deciding to.
Block rate over time
2 daysThe percentage of domains in the corpus whose robots.txt disallows Google-Extended, measured once per domain per pass. The denominator is how many domains were read that day and is stored with each point, so the line is not restated when the corpus grows.
The other seven
- OpenAI
- GPTBot
- OpenAI
- OAI-SearchBot
- Anthropic
- ClaudeBot
- Perplexity
- PerplexityBot
- Apple
- Applebot-Extended
- Common Crawl
- CCBot
- ByteDance
- Bytespider
How this page is made
The explanation above is written by a person. The numbers are computed from Domduck’s own daily reads of each domain’s robots.txt, parsed to RFC 9309 rather than searched for a string: a rule binds to the nearest preceding group of user-agent lines, the longest matching pattern wins, and Allow breaks a tie. No prefix matching is done, so a group naming Google is not treated as a policy about Google-Extended. What the collectors have read.