AI crawler policy

Domduck reads every tracked domain's robots.txt once a day and records what it says about eight named AI crawlers, plus whether the domain serves an llms.txt.

This is a reputation surface that moves without anyone deciding to move it. A CDN ships a new default, a security team pastes a blocklist, a migration drops the file, and the domain quietly leaves the AI-visible web. Nothing tells the owner. A dated daily reading turns that into an event.

The eight agents

Google-Extended is the one worth reading twice. Blocking it takes a site out of Gemini and out of AI Overviews grounding, and it does not affect normal Google Search ranking. People block it believing it does.

Three states, not two

Every agent carries two verdicts.

**unspecified is not allowed.** A file with User-agent: * and Disallow: / and no AI agent named anywhere scores unspecified for all eight and blocks all eight. Read effective for what happens; read named for what the site actually wrote down. A tool that reports one boolean per agent is giving the wrong answer for exactly the sites where the question matters.

How the file is read

The parser follows RFC 9309 rather than searching the text for a name.

• Disallow binds to the nearest preceding run of User-agent lines, and a new User-agent after a rule line starts a new group.
• One group can name several agents.
• Matching is case-insensitive.
• The longest matching path pattern wins, and Allow breaks a tie.
• No prefix matching. User-agent: Google is not a statement about Google-Extended. Reporting a policy the site never wrote is worse than reporting none.

A file that names an agent and then allows it would come back "blocked" from a naive substring search, which is the wrong answer to the only question this source is asked.

Other things recorded on the same pass: the HTTP status, whether there is a file at all (no file means no policy, so every crawler is welcome), a redirect that left the domain, whether the file was too large to read whole, a hash of the bytes, and the llms.txt probe.

Reading it from an agent

get_ai_crawler_policy is on both the authenticated and the public server.

curl -s https://domduck.com/api/mcp/public \
  -H "content-type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
       "params":{"name":"get_ai_crawler_policy","arguments":{"domain":"stripe.com"}}}'

The numeric series behind it is robots_txt, which counts how many of the 8 agents are effectively blocked. Lower is better, and it is the one series in Domduck that a site owner can change in an afternoon.

llms.txt

Recorded as present or absent on the same daily pass, from one extra GET to /llms.txt.

Be honest about what it buys. There is no evidence that any search crawler reads it unprompted, and the sensible model is that it gets read when an agent is pointed at a domain rather than discovered by a training crawler. Domduck serves one because a tool that measures llms.txt adoption and does not have one is making an argument it does not follow.