Learn

Plain definitions for everything Domduck measures, and what each reading actually claims.

24 entries and 6 guides. Each one says what the term means, what Domduck records about it and how often, and what the number does not claim. Where a source is built and not collecting, the entry says so and /status shows the row.

AI access

Which crawlers a site lets in, the files that say so, and the three answers each file can give.

AI crawlers

AI crawlers are the agents that fetch web pages to train models or to cite them in answers. Eight of them matter, and they are eight separate decisions.

llms.txt

llms.txt is a proposed markdown file at a site’s root that tells a language model what the site is and which pages to read first.

  • robots.txtrobots.txt is a file at the root of a site that tells crawlers which paths they may fetch, one group of rules per user agent.
  • Allowed, disallowed, unspecifiedUnspecified is the third answer a robots.txt file can give about a crawler: the file exists, and it never mentions that agent at all.
  • Training crawlers vs retrieval crawlersA training crawler collects pages to build a model. A retrieval crawler fetches a page so an answer can cite it. Blocking one does nothing to the other.
  • llms.txt vs robots.txtrobots.txt controls access and crawlers obey it. llms.txt suggests what to read and nothing enforces it. The two files answer different questions.
  • llms-full.txtllms-full.txt is a convention for one file holding a whole documentation set as text. It is not part of the llms.txt proposal.
  • llms.txt vs MCPllms.txt is a file a model reads. MCP is a protocol a model calls. One describes a site, the other answers questions about it.

Authority

The scores that try to say how much weight a domain carries, and what each one measures.

Domain Rating

Domain Rating is Ahrefs’ 0 to 100 score for the strength of a domain’s backlink profile. It is logarithmic, relative, and it is not a Google metric.

  • Domain AuthorityDomain Authority is Moz’s 1 to 100 score predicting how well a domain ranks. It is a prediction from Moz’s link index, not a measurement from Google.
  • URL RatingURL Rating is Ahrefs’ 0 to 100 score for a single page rather than a whole domain. Domain Rating is the site-wide one.
  • OpenPageRankOpenPageRank is a free 0 to 10 domain score built from the Common Crawl link graph by DomCop, modelled on Google’s original PageRank.
  • Domain trust scoreA domain trust score is any vendor’s own 0 to 100 judgment of whether a domain is trustworthy. There is no standard behind the phrase.

Identity

The registration record: who holds the name, since when, and which protocol answers the question.

Domain age

Domain age is the time since a domain was first registered, read from the public registration record rather than from when the site went live.

  • RDAPRDAP is the Registration Data Access Protocol: a JSON web API that returns who registered a domain and when, replacing WHOIS.
  • WHOISWHOIS is the 1980s text protocol on port 43 that returned domain registration details, now replaced by RDAP for generic top-level domains.
  • RDAP vs WHOISRDAP returns structured JSON over HTTPS with a defined schema. WHOIS returned free text over port 43 in whatever shape each registry chose.
  • RDAP bootstrapRDAP bootstrap is how a client finds the right server for a domain: fetch IANA’s registry, match the top-level domain, get the base URL.
  • Domain status codesDomain status codes are the EPP labels a registry returns for a name, such as clientTransferProhibited or pendingDelete, that say what may happen to it next.

Infrastructure

Certificates and DNS. The parts that break on a Thursday and take the site with them.

  • TLS certificateA TLS certificate is the signed file a site presents to prove it owns its name, naming the domains it covers and the dates it is valid between.
  • SAN certificateA SAN certificate lists every hostname it covers in its Subject Alternative Name field, which is the only field browsers still read for name matching.
  • SSL certificate expiryA certificate expires at a fixed timestamp written inside it. After that moment every browser and every client refuses the connection.
  • Certificate TransparencyCertificate Transparency is a set of public append-only logs recording every certificate a trusted authority issues, so misissuance can be found.
  • DNS recordsDNS records are the entries that map a domain to addresses and services: A and AAAA for hosts, MX for mail, NS for the authoritative servers.

Popularity

Ranking lists after Alexa: what each one measures, and where each one stops.

Website popularity rank

A website popularity rank orders domains by how much of the web’s attention they get. Every current list measures a different proxy for it.

  • TrancoTranco is a research ranking of the top million domains, built by averaging several source lists over 30 days so single-day spikes cannot move it.
  • Alexa rankAlexa rank was Amazon’s traffic ranking of websites. The service closed on 2022-05-01 and its APIs followed on 2022-12-15.
  • Cloudflare Radar rankCloudflare Radar ranks domains by resolver query volume. It orders the top 100 exactly and puts everything below into buckets.

Reputation

One phrase, two subjects. Which one you meant decides which tool you need.

Domain reputation

Domain reputation means two different things: whether your mail gets delivered, and how much weight your domain carries in search.

Method

How Domduck reads a domain, and what a gap in a chart means.

How Domduck reads a domain

Domduck writes one dated row per check and never overwrites one, so a gap in a chart means nobody looked rather than nothing happened.

The eight crawlers

Each agent has its own page with a dated block-rate series behind it, under /ai-crawlers. They are not repeated here, because one query deserves one page and splitting it would give each half the data.