# AI crawler policy

> The eight agents Domduck tracks, the three states each one can be in, and why unspecified is not the same as allowed.

Domduck reads every tracked domain's `robots.txt` once a day and
records what it says about eight named AI crawlers, plus whether the domain
serves an `llms.txt`.

This is a reputation surface that moves without anyone deciding to move it. A
CDN ships a new default, a security team pastes a blocklist, a migration drops
the file, and the domain quietly leaves the AI-visible web. Nothing tells the
owner. A dated daily reading turns that into an event.

## The eight agents

| Agent | Operator | Blocking it means |
|---|---|---|
| `GPTBot` | OpenAI | No ChatGPT browsing or training |
| `OAI-SearchBot` | OpenAI | No ChatGPT search surface |
| `ClaudeBot` | Anthropic | No Claude retrieval |
| `PerplexityBot` | Perplexity | No Perplexity citation |
| `Google-Extended` | Google | Out of Gemini and AI Overviews grounding, Search ranking unaffected |
| `Applebot-Extended` | Apple | Out of Apple Intelligence training |
| `CCBot` | Common Crawl | Out of most open training corpora |
| `Bytespider` | ByteDance | Out of TikTok and Doubao |

`Google-Extended` is the one worth reading twice. Blocking it takes a site out
of Gemini and out of AI Overviews grounding, and it does **not** affect normal
Google Search ranking. People block it believing it does.

## Three states, not two

Every agent carries two verdicts.

| Field | Meaning |
|---|---|
| `named` | What the file spells out about this agent: `allowed`, `disallowed`, or `unspecified` when no group names it. |
| `effective` | What is actually in force: the named group if there is one, otherwise the catch-all, otherwise allowed. |

**`unspecified` is not `allowed`.** A file with `User-agent: *` and
`Disallow: /` and no AI agent named anywhere scores `unspecified` for all
eight and blocks all eight. Read `effective` for what happens; read `named`
for what the site actually wrote down. A tool that reports one boolean per agent
is giving the wrong answer for exactly the sites where the question matters.

## How the file is read

The parser follows RFC 9309 rather than searching the text for a name.

- `Disallow` binds to the nearest preceding run of `User-agent` lines, and a
  new `User-agent` after a rule line starts a new group.
- One group can name several agents.
- Matching is case-insensitive.
- The longest matching path pattern wins, and `Allow` breaks a tie.
- **No prefix matching.** `User-agent: Google` is not a statement about
  `Google-Extended`. Reporting a policy the site never wrote is worse than
  reporting none.

A file that names an agent and then allows it would come back "blocked" from a
naive substring search, which is the wrong answer to the only question this
source is asked.

Other things recorded on the same pass: the HTTP status, whether there is a file
at all (no file means no policy, so every crawler is welcome), a redirect that
left the domain, whether the file was too large to read whole, a hash of the
bytes, and the `llms.txt` probe.

## Reading it from an agent

`get_ai_crawler_policy` is on both the [authenticated](https://domduck.com/docs/mcp) and the
[public](https://domduck.com/docs/mcp-public) server.

```bash
curl -s https://domduck.com/api/mcp/public \
  -H "content-type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
       "params":{"name":"get_ai_crawler_policy","arguments":{"domain":"stripe.com"}}}'
```

The numeric series behind it is `robots_txt`, which counts how many of the
8 agents are effectively blocked. **Lower is better**, and
it is the one series in Domduck that a site owner can change in an afternoon.

## llms.txt

Recorded as present or absent on the same daily pass, from one extra `GET` to
`/llms.txt`.

Be honest about what it buys. There is no evidence that any search crawler reads
it unprompted, and the sensible model is that it gets read when an agent is
pointed at a domain rather than discovered by a training crawler. Domduck
[serves one](https://domduck.com/llms.txt) because a tool that measures llms.txt
adoption and does not have one is making an argument it does not follow.