CCBot
The crawler behind Common Crawl, a free public archive of the web that most open training corpora are built from.
Measured across 58 domains read on 12 October 2026, against 14.1% on 2 August 2026.
disallow it
What blocking CCBot costs
The largest indirect one in the table. Common Crawl is an input to many models rather than one, so blocking CCBot removes you from a corpus you cannot enumerate, and it does so for every future model built on it.
What it does not cost
Archives already published keep the pages they already hold. Common Crawl does not retract past crawls.
It is also the oldest entry here, so a robots.txt written years ago for reasons unrelated to AI may already block it. Domduck reports what the file says today, not why it says it.
User-agent: CCBot
Disallow: /User-agent: CCBot
Disallow:Allowed, and said out loud.
Sites that changed their mind
last 90 days- 7 October 2026 (2026-10-07)
- ap.orgCCBot was blocked between 15 September 2026 and 7 October 2026.
- 5 October 2026 (2026-10-05)
- allmusic.comCCBot was unblocked between 21 September 2026 and 5 October 2026.
- thelancet.comCCBot was unblocked between 15 September 2026 and 5 October 2026.
- 3 October 2026 (2026-10-03)
- kniga-online.comCCBot was unblocked between 15 September 2026 and 3 October 2026.
- aflamk1.netCCBot was unblocked between 15 September 2026 and 3 October 2026.
- 2 October 2026 (2026-10-02)
- dorama.landCCBot was unblocked between 15 September 2026 and 2 October 2026.
- 1 October 2026 (2026-10-01)
- depositphotos.comCCBot was blocked between 17 September 2026 and 1 October 2026.
- 30 September 2026 (2026-09-30)
- gemidos.tvCCBot was unblocked between 15 September 2026 and 30 September 2026.
- getmodsapk.comCCBot was unblocked between 15 September 2026 and 30 September 2026.
- apuestasroyal.comCCBot was unblocked between 15 September 2026 and 30 September 2026.
- porn-comic.comCCBot was unblocked between 15 September 2026 and 30 September 2026.
- bigbadtoystore.comCCBot was unblocked between 21 August 2026 and 30 September 2026.
- vid30s.comCCBot was unblocked between 15 September 2026 and 30 September 2026.
- distintas.netCCBot was unblocked between 15 September 2026 and 30 September 2026.
- romeo.comCCBot was unblocked between 15 September 2026 and 30 September 2026.
- insidehighered.comCCBot was unblocked between 15 September 2026 and 30 September 2026.
- livecamrips.toCCBot was unblocked between 15 September 2026 and 30 September 2026.
- 29 September 2026 (2026-09-29)
- emojicombos.comCCBot was unblocked between 15 September 2026 and 29 September 2026.
- obqj2.comCCBot was unblocked between 15 September 2026 and 29 September 2026.
- momvids.comCCBot was unblocked between 15 September 2026 and 29 September 2026.
The other seven
OpenAI’s crawler for gathering web content used to train its models.
The crawler behind ChatGPT’s search surface: it fetches pages so they can be cited in answers.
Anthropic’s crawler, used to gather web content for Claude.
Perplexity’s crawler, which indexes pages so they can be cited in its answers.
Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.
The same shape as Google-Extended: a control token, not a crawler. Applebot fetches the page either way; this decides whether the content may train Apple Intelligence.
ByteDance’s crawler, gathering content used for TikTok’s search and for Doubao.
How this page is made
The explanation above is written by a person. The numbers are computed from Domduck’s own daily reads of each domain’s robots.txt, parsed to RFC 9309 rather than searched for a string: a rule binds to the nearest preceding group of user-agent lines, the longest matching pattern wins, and Allow breaks a tie. No prefix matching is done, so a group naming Google is not treated as a policy about Google-Extended. What the collectors have read.