Allowed, disallowed, unspecified

Unspecified is the third answer a robots.txt file can give about a crawler. The file exists, it has rules, and it never mentions that agent anywhere. Nearly every AI crawler checker collapses this into "allowed", and that is wrong in two different ways.

Why unspecified is not allowed

First, the catch-all may already block it. A file with User-agent: * and Disallow: / blocks every agent that has no group of its own, so an agent that is never named is blocked rather than allowed. A checker that reads only "is GPTBot mentioned" returns the opposite of the truth on exactly those sites.

Second, the two states have different futures. An agent allowed by name stays allowed when the site adds a catch-all rule next month. An agent that is not named gets caught by it silently, on a day nobody remembers.

What Domduck records

Every reading carries both fields per agent: named, which is what the file spells out about that agent, and effective, which is the verdict in force once the catch-all is applied. The catch-all group is stored beside them.

On 2026-08-03, across 2,228 domains, GPTBot was blocked on 17.0%, allowed by name on 5.4%, and not named at all on 80.4%. The third column is the biggest one for all eight agents, and it is the column nobody else publishes. Current figures are on the AI crawlers page.

Why it matters

A number that merges two states cannot be compared with itself later. If "allowed" means both "the site said yes" and "the site never thought about it", then a rise in the allowed rate could be adoption or it could be silence, and no amount of later analysis separates them.

That is also why Domduck's numeric series for this source counts blocked agents rather than allowed ones. Blocked is a decision. Allowed, merged, is not.