robots.txt

robots.txt is a file at the root of a site that tells crawlers which paths they may fetch, one group of rules per user agent.

robots.txt is a file at the root of a site that tells crawlers which paths they may fetch. It is one plain text file, at /robots.txt and nowhere else, holding groups of rules. Each group starts with one or more User-agent lines and continues with Allow and Disallow lines that apply to those agents.

The format was folklore for 25 years and became a standard in 2022, as RFC

  1. The rules that matter and that most parsers get wrong:
  • A Disallow line belongs to the nearest run of User-agent lines above it. A new User-agent after a rule line starts a new group.
  • The longest matching pattern wins, and Allow breaks a tie against a Disallow of the same length.
  • Matching on the agent name is exact, not by prefix. User-agent: Google is not a rule about Google-Extended.
  • User-agent: * is the catch-all group, and it applies only to agents that have no group of their own.

What Domduck records

The robots_txt source reads GET /robots.txt once a day for tracked domains and on a slower cadence across the seeded corpus. Each reading stores the HTTP status, whether the file exists at all, a verdict for each of the eight tracked AI crawlers in three states, the catch-all group beside them, and a sha256 of the bytes.

The request sends the previous reading's ETag. Most files do not change, so the answer is usually a 304 and the prior verdicts carry forward with a fresh timestamp. A day with no reading at all is stored as a gap rather than as a repeat, because unchecked is not unchanged.

Why it matters

A missing robots.txt is not the same as an empty one, and neither is the same as a file that allows everything. Domduck records absent separately for that reason.

The file is also advisory. It declares a position and it is not a control. Well behaved crawlers honor it, badly behaved ones read it and ignore it, and published measurement puts the gap at around 39.5% of GPTBot bans not enforced. A tool that reports robots.txt as if it were traffic is reporting the wrong thing, so Domduck says "declared" everywhere it means declared.

One more property makes the file worth recording daily: it has no history. Change it and the old version is gone. See the live block rates, read on 2026-08-03 across 2,228 domains.

Related

  • Allowed, disallowed, unspecifiedUnspecified is the third answer a robots.txt file can give about a crawler: the file exists, and it never mentions that agent at all.
  • llms.txt vs robots.txtrobots.txt controls access and crawlers obey it. llms.txt suggests what to read and nothing enforces it. The two files answer different questions.
  • Training crawlers vs retrieval crawlersA training crawler collects pages to build a model. A retrieval crawler fetches a page so an answer can cite it. Blocking one does nothing to the other.

Primary source: RFC 9309, the Robots Exclusion Protocol