The AI Crawler Index
What the web tells eight AI crawlers, measured from robots.txt across the corpus and published with the date every reading was taken.
Last updated
Eight AI crawlers decide whether your writing reaches a generative answer. This index reports what the web's robots.txt files say about each one, across the domains Domduck read on 2026-08-03.
Sample: 2,228 domains. Series: 2026-08-02 to 2026-08-03, 2 days of readings.
What the web says
| Agent | Operator | Blocked | Allowed by name | Not named | 30-day change |
|---|---|---|---|---|---|
| GPTBot | OpenAI | 17.0% | 5.4% | 80.4% | n/a |
| OAI-SearchBot | OpenAI | 6.7% | 5.0% | 91.2% | n/a |
| ClaudeBot | Anthropic | 16.1% | 4.0% | 83.1% | n/a |
| PerplexityBot | Perplexity | 10.6% | 4.8% | 87.4% | n/a |
| Google-Extended | 14.5% | 4.3% | 84.2% | n/a | |
| Applebot-Extended | Apple | 13.6% | 1.6% | 88.2% | n/a |
| CCBot | Common Crawl | 17.5% | 2.6% | 83.3% | n/a |
| Bytespider | ByteDance | 16.3% | 1.8% | 85.5% | n/a |
There is no change column yet. The series starts on 2026-08-02 and a month-over-month figure needs a reading about a month back. Comparing against the oldest day available and calling it a month would be a made-up number.
Most blocked on this reading: CCBot at 17.5%. Least blocked: OAI-SearchBot at 6.7%.
4.0% of the corpus carries a catch-all Disallow: /, which blocks every agent that is not named in a group of its own. 4.8% serve an llms.txt.
The third column is the finding
"Not named" is the majority answer for every agent, and it is not a rounding of "allowed".
A file that never mentions an agent has not allowed it. A catch-all rule blocks it today if one exists, and a catch-all added next month catches it silently. The two states also have different futures, which is why merging them produces a number that cannot be compared with itself later: a rise in "allowed" could be adoption or could be silence.
Nearly every published AI crawler checker returns a boolean. On this corpus that boolean is wrong for the majority of domains, in one direction or the other.
What this data cannot show
Four limits, stated before anyone finds them.
- robots.txt is a declaration, not a control. It records what a site publishes, never what a crawler does. Published measurement puts unenforced GPTBot bans near 39.5%.
- A network-level block is invisible here. A site can go dark to an agent at its firewall or its CDN with no change to robots.txt, and no reader of that file can see it. Detecting it needs a fetch sent as the agent, which is a different source and is not part of this index.
- No intent is inferred anywhere. This index says an agent was blocked on a date. It never says a site decided to block AI, because the file does not say that and neither can we.
- The corpus is not the web. It is a seeded set of domains plus every domain a person has looked up. Every rate here is a rate within that set, and the sample size sits beside every figure for that reason.
Method
The corpus is seeded from a public top-sites list and grows with the domains people look up. Each domain's /robots.txt is fetched on a schedule, and the response is parsed against RFC 9309 rather than searched for a string: a Disallow binds to the nearest preceding run of User-agent lines, a new User-agent after a rule line starts a new group, the longest matching pattern wins, and Allow breaks ties.
Agent matching is exact. User-agent: Google is not treated as a rule about Google-Extended, because reporting a policy a site never wrote is worse than reporting none.
Each reading carries two verdicts per agent: what the file names, and what is in force once the catch-all is applied. The published rates use the second.
Requests are conditional. The previous reading's ETag goes out with each fetch, so an unchanged file answers 304 and the prior verdicts carry forward with a fresh timestamp. A day with no reading is stored as a gap rather than as a repeat.
Denominators are frozen into each row. The corpus grows, so every historic rate stays computed against the number of domains read on that day, and no later pass can restate a published figure.
Date precision follows the cadence. Domains people track are read daily. Seeded domains are read on a slower rotation, so a change on one of those is dated to the pass that found it rather than to the day it happened.
The parser's rules and the three states it reports are documented at robots.txt, and it is being extracted into a standalone package so the measurement can be checked rather than trusted.
The data
The full aggregate series, one row per day per agent, is at /research/ai-crawler-index.csv. It carries no domain names and no account data. It is free to use with attribution to Domduck and a link to this page.
Columns: day, agent, domains_read, blocked, allowed_by_name, unspecified.
What comes next
Cloudflare changes its AI-crawler default on 2026-09-15, for new customers, new sites of existing customers, and existing free-tier sites. A large number of sites will change posture on one day without anyone editing a file.
Robots.txt has no history. Change it and the previous version is gone, so what a site allowed on 14 September can only be known if a third party wrote it down first. This index exists to be that record, and the reason it is running now rather than in September is that it cannot be started afterwards.
Editions: this index monthly, a longer cut quarterly, and a full-year reference with the complete series.
Use it
Any figure here is free to quote with attribution. If you need a cut this page does not show, by top-level domain, by rank band, by registrar or by date, write to [[email protected]](mailto:[email protected]) and it gets run.
The live table updates as readings land, the change feed lists policy changes as they are observed, and how Domduck reads a domain explains the mechanics.