llms.txt vs robots.txt
robots.txt controls access and crawlers obey it. llms.txt suggests what to read and nothing enforces it. The two files answer different questions.
robots.txt controls access and crawlers obey it. llms.txt suggests what to read and nothing enforces it. The two files sit at the same place in a site's root and answer different questions.
| robots.txt | llms.txt | |
|---|---|---|
| Age | folklore since 1994, standard since 2022 (RFC 9309) | proposed 2024-09-03 |
| Format | groups of User-agent and Disallow rules | markdown: a title, a summary, then lists of links with notes |
| Question it answers | may you fetch this path | what should you read first, and where is the clean copy |
| Enforcement | honored by well behaved crawlers, ignored by others | none. It is a hint. |
| Read by search engines | yes | Google has said Search does not use it |
What llms.txt actually is
A markdown file at /llms.txt. The proposal defines one required part, an H1 with the site name, then an optional blockquote summary, then H2 sections holding lists of links in the form [name](url): note. An Optional H2 has a defined meaning: those links may be skipped when a shorter context is needed.
The proposal's second half gets forgotten and it is the useful half: serve each page as clean markdown at the same URL with .md appended. Domduck does that at /docs/mcp.md and everywhere else under /docs.
What Domduck records
The same daily pass that reads robots.txt probes for llms.txt and stores whether it is present. Presence only, not the contents, so the reading is one boolean with a date on it and the cost is one conditional request.
On 2026-08-03, 4.76% of the 2,228 domains read that day served an llms.txt. The live figure is on the AI crawlers page.
Why it matters
Do not put access rules in llms.txt. Nothing enforces them, and a site that writes "do not train on this" there has published a preference, not a control. The control is robots.txt, and even that is advisory.
Do not expect llms.txt to move rankings either. Ahrefs measured 97% of 137,000 llms.txt files receiving zero requests. The realistic model is that the file gets read when an agent is pointed at your domain, which is worth something if your audience runs agents and worth nothing otherwise.
Related
- robots.txtrobots.txt is a file at the root of a site that tells crawlers which paths they may fetch, one group of rules per user agent.
- llms-full.txtllms-full.txt is a convention for one file holding a whole documentation set as text. It is not part of the llms.txt proposal.
- llms.txt vs MCPllms.txt is a file a model reads. MCP is a protocol a model calls. One describes a site, the other answers questions about it.
Primary source: llmstxt.org, the proposal