AI crawlers

AI crawlers are the agents that fetch web pages to train models or to cite them in answers. Domduck tracks eight of them, and the first thing worth saying is that they are eight decisions rather than one.

Most published advice treats "block AI" as a single switch. It is not. Blocking a training crawler and blocking a citation crawler cost different things, and two of the eight names are not crawlers at all.

The eight, and what blocking each one costs

Google-Extended is the one people get wrong. Blocking it does not affect Search ranking, and almost everyone believes it does.

Three states, not two

A robots.txt file gives three answers about an agent, not two: allowed by name, blocked, or never mentioned. The third one is not a rounding of the first. A file that never names GPTBot has not allowed it, because a catch-all rule blocks it anyway and because a catch-all added next month catches it silently.

On 2026-08-03, across 2,228 domains, GPTBot was blocked on 17.0%, allowed by name on 5.4%, and not named at all on 80.4%. That third column is the majority answer for all eight agents, and it is the one every boolean checker reports as "allowed".

What a robots.txt rule can and cannot do

It declares a position. It is not a control.

• Well behaved crawlers honor it. Others read it and continue. Published measurement puts the gap at around 39.5% of GPTBot bans not enforced.
• Blocking after the fact removes nothing. Content already collected stays collected.
• A block at the network or firewall layer is invisible to a robots.txt reader, including Domduck. A site can go dark to an agent without its file changing, and no robots.txt parser can see that.

What Domduck records

One reading a day per domain for tracked domains, and a slower pass across the seeded corpus. Each reading stores the file's status, the verdict per agent in three states, the catch-all beside them, the llms.txt presence, and a hash of the bytes.

The reason for the daily cadence is the one fact behind this whole section: robots.txt has no history. Change it and the previous file is gone. There is no archive at the fetch level and the site has no reason to keep one, so the date a policy changed exists only if somebody was reading the file before the change.

The live corpus figures are on the AI crawlers page, and the policy changes as they happen are in the change feed.

How to decide

1. Decide about citation first. That is OAI-SearchBot and PerplexityBot. If you want the readers, allow them, and blocking the training crawlers does not affect this.
2. Decide about training second. GPTBot, ClaudeBot, CCBot, Bytespider. This is a values and licensing question, and it is about the future rather than the past.
3. Name every agent you have an opinion about. Silence is not a decision, and it is not recorded as one.
4. Write the file, then check it from outside. A rule with a typo in the agent name is not a rule.