Learn
Plain definitions for everything Domduck measures, and what each reading actually claims.
Each one says what the term means, what Domduck records about it and how often, and what the number does not claim. Where a source is built and not collecting, the entry says so and /status shows the row.
- AI crawlers
AI crawlers are the agents that fetch web pages to train models or to cite them in answers. Eight of them matter, and they are eight separate decisions.
- llms.txt
llms.txt is a proposed markdown file at a site’s root that tells a language model what the site is and which pages to read first.
- Domain Rating
Domain Rating is Ahrefs’ 0 to 100 score for the strength of a domain’s backlink profile. It is logarithmic, relative, and it is not a Google metric.
- Domain age
Domain age is the time since a domain was first registered, read from the public registration record rather than from when the site went live.
- Website popularity rank
A website popularity rank orders domains by how much of the web’s attention they get. Every current list measures a different proxy for it.
- Domain reputation
Domain reputation means two different things: whether your mail gets delivered, and how much weight your domain carries in search.
- How Domduck reads a domain
Domduck writes one dated row per check and never overwrites one, so a gap in a chart means nobody looked rather than nothing happened.
By subject
31 pagesAI access
Which crawlers a site lets in, the files that say so, and the three answers each file can give.
AI crawlers
AI crawlers are the agents that fetch web pages to train models or to cite them in answers. Eight of them matter, and they are eight separate decisions.
llms.txt
llms.txt is a proposed markdown file at a site’s root that tells a language model what the site is and which pages to read first.
Authority
The scores that try to say how much weight a domain carries, and what each one measures.
Domain Rating
Domain Rating is Ahrefs’ 0 to 100 score for the strength of a domain’s backlink profile. It is logarithmic, relative, and it is not a Google metric.
Identity
The registration record: who holds the name, since when, and which protocol answers the question.
Domain age
Domain age is the time since a domain was first registered, read from the public registration record rather than from when the site went live.
Infrastructure
Certificates and DNS. The parts that break on a Thursday and take the site with them.
Popularity
Ranking lists after Alexa: what each one measures, and where each one stops.
Website popularity rank
A website popularity rank orders domains by how much of the web’s attention they get. Every current list measures a different proxy for it.
Reputation
One phrase, two subjects. Which one you meant decides which tool you need.
Domain reputation
Domain reputation means two different things: whether your mail gets delivered, and how much weight your domain carries in search.
Method
How Domduck reads a domain, and what a gap in a chart means.
How Domduck reads a domain
Domduck writes one dated row per check and never overwrites one, so a gap in a chart means nobody looked rather than nothing happened.
The eight crawlers
not repeated hereEach agent has its own page with a dated block-rate series behind it, under /ai-crawlers. They are not repeated here, because one query deserves one page and splitting it would give each half the data.
OpenAI’s crawler for gathering web content used to train its models.
The crawler behind ChatGPT’s search surface: it fetches pages so they can be cited in answers.
Anthropic’s crawler, used to gather web content for Claude.
Perplexity’s crawler, which indexes pages so they can be cited in its answers.
Not a crawler at all. It is a control token: Googlebot still fetches the page, and this decides whether the content may be used for Gemini and for grounding AI Overviews.
The same shape as Google-Extended: a control token, not a crawler. Applebot fetches the page either way; this decides whether the content may train Apple Intelligence.
The crawler behind Common Crawl, a free public archive of the web that most open training corpora are built from.
ByteDance’s crawler, gathering content used for TikTok’s search and for Doubao.