OAI-SearchBot
The crawler behind ChatGPT’s search surface: it fetches pages so they can be cited in answers.
Measured across 58 domains read on 12 October 2026, against 5.1% on 2 August 2026.
disallow it
What blocking OAI-SearchBot costs
Your pages stop appearing as citations in ChatGPT search results. For most publishers this is the expensive one to block, because a citation is a link with a reader attached.
What it does not cost
It is not a training crawler. Blocking it does not keep your content out of a model, and allowing it does not put your content into one.
This is the agent to think hardest about. The traffic case for allowing it is the strongest of the eight, and the training objection does not apply to it.
User-agent: OAI-SearchBot
Disallow: /User-agent: OAI-SearchBot
Disallow:Allowed, and said out loud.
Sites that changed their mind
last 90 days- 10 October 2026 (2026-10-10)
- joongang.co.krOAI-SearchBot was unblocked between 26 September 2026 and 10 October 2026.
- lynk.idOAI-SearchBot was unblocked between 26 September 2026 and 10 October 2026.
- 3 October 2026 (2026-10-03)
- claude.aiOAI-SearchBot was unblocked on 3 October 2026.
- 2 October 2026 (2026-10-02)
- claude.aiOAI-SearchBot was blocked on 2 October 2026.
- 29 September 2026 (2026-09-29)
- utraff.comOAI-SearchBot was unblocked between 15 September 2026 and 29 September 2026.
- rutracker.orgOAI-SearchBot was blocked between 15 September 2026 and 29 September 2026.
- oneindia.comOAI-SearchBot was unblocked between 15 September 2026 and 29 September 2026.
- sportfilm800.comOAI-SearchBot was blocked between 15 September 2026 and 29 September 2026.
- gsmarena.comOAI-SearchBot was unblocked between 15 September 2026 and 29 September 2026.
- 26 September 2026 (2026-09-26)
- newspapers.comOAI-SearchBot was blocked between 31 August 2026 and 26 September 2026.
- 11st.co.krOAI-SearchBot was unblocked between 4 September 2026 and 26 September 2026.
- claude.aiOAI-SearchBot was unblocked on 26 September 2026.
- 16 September 2026 (2026-09-16)
- slrclub.comOAI-SearchBot was unblocked between 1 September 2026 and 16 September 2026.
- ssg.comOAI-SearchBot was unblocked between 31 August 2026 and 16 September 2026.
- 15 September 2026 (2026-09-15)
- wikicv.netOAI-SearchBot was blocked between 31 August 2026 and 15 September 2026.
- backblazeb2.comOAI-SearchBot was blocked between 31 August 2026 and 15 September 2026.
- screenrant.comOAI-SearchBot was unblocked between 31 August 2026 and 15 September 2026.
- ad-score.comOAI-SearchBot was blocked between 31 August 2026 and 15 September 2026.
- 2 September 2026 (2026-09-02)
- mix.comOAI-SearchBot was blocked between 18 August 2026 and 2 September 2026.
- 1 September 2026 (2026-09-01)
- theculturetrip.comOAI-SearchBot was blocked between 18 August 2026 and 1 September 2026.
The other seven
OpenAI’s crawler for gathering web content used to train its models.
Anthropic’s crawler, used to gather web content for Claude.
Perplexity’s crawler, which indexes pages so they can be cited in its answers.
Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.
The same shape as Google-Extended: a control token, not a crawler. Applebot fetches the page either way; this decides whether the content may train Apple Intelligence.
The crawler behind Common Crawl, a free public archive of the web that most open training corpora are built from.
ByteDance’s crawler, gathering content used for TikTok’s search and for Doubao.
How this page is made
The explanation above is written by a person. The numbers are computed from Domduck’s own daily reads of each domain’s robots.txt, parsed to RFC 9309 rather than searched for a string: a rule binds to the nearest preceding group of user-agent lines, the longest matching pattern wins, and Allow breaks a tie. No prefix matching is done, so a group naming Google is not treated as a policy about Google-Extended. What the collectors have read.