# Data sources

> Every source Domduck reads, what its numbers mean, which direction is better, and how often it is re-read.

Every source Domduck reads, and what its numbers mean.

| Source | Reads | Cadence |
|---|---|---|
| `ahrefs_dr` | Domain Rating. 0-100, higher is better. | daily |
| `tranco` | Tranco rank. top 1M, lower is better. | daily |
| `openpagerank` | OpenPageRank. 0-10, higher is better. | daily |
| `cloudflare_radar` | Cloudflare Radar. rank plus content categories. | daily |
| `rdap` | Registration. registration date and status. | once |
| `self_tls` | TLS certificate. issuer, validity, days remaining. | daily |
| `self_dns` | DNS. A, AAAA, MX, NS, SPF, DMARC, DNSSEC. | daily |
| `robots_txt` | AI crawler policy. count of 8 tracked AI crawlers blocked, lower is better. | daily |

## Direction is not obvious, so it is stated

`tranco`, `cloudflare_radar` and `robots_txt` are **better when lower**.
`ahrefs_dr` and `openpagerank` are better when higher. Every history the MCP
server returns carries a `higherIsBetter` field for this reason: a rank series
read by something that assumes bigger is better is a chart drawn upside down.

## Every reading is a dated observation

Nothing is ever overwritten. A reading is appended with the timestamp it was
taken at, and the current value is a query over that history rather than a
column.

The consequence worth knowing before you read a series: **a missing day means
nobody looked.** It does not mean the value was zero and it does not mean the
value held. That distinction is why the charts draw a gap for a day with no
reading and a dashed line for a value carried past the last reading, and it is
why `null` appears in an API response where a lazier design would send `0`.

## Notes per source

- **`ahrefs_dr`** Domain Rating, 0-100, on a logarithmic scale, so 80 to 81 is
  a much larger move than 20 to 21.
- **`tranco`** A research list that averages 30 days of several popularity
  rankings, so it is far more stable than any single day's list. Only the top 1M
  exists; outside it, there is correctly no reading rather than a bad one.
- **`openpagerank`** 0-10. Free, coarse, and useful mainly as a second opinion
  when Domain Rating moves.
- **`cloudflare_radar`** An exact rank for roughly the top 100 domains and a
  bucket ceiling below that, plus content categories.
- **`rdap`** Registration date, registrar and status, from the registry through
  IANA's bootstrap. Read once: it does not change.
- **`self_tls`** Our own TLS handshake with the domain. Issuer, validity window
  and days remaining.
- **`self_dns`** A, AAAA, MX, NS, SPF, DMARC and DNSSEC, through a fixed DNS
  over HTTPS resolver.
- **`robots_txt`** The [AI crawler policy](https://domduck.com/docs/ai-crawlers), plus an
  `llms.txt` presence probe on the same pass.

## The eight AI crawlers

| Agent | Operator | Blocking it means |
|---|---|---|
| `GPTBot` | OpenAI | No ChatGPT browsing or training |
| `OAI-SearchBot` | OpenAI | No ChatGPT search surface |
| `ClaudeBot` | Anthropic | No Claude retrieval |
| `PerplexityBot` | Perplexity | No Perplexity citation |
| `Google-Extended` | Google | Out of Gemini and AI Overviews grounding, Search ranking unaffected |
| `Applebot-Extended` | Apple | Out of Apple Intelligence training |
| `CCBot` | Common Crawl | Out of most open training corpora |
| `Bytespider` | ByteDance | Out of TikTok and Doubao |

Full detail on [the AI crawler page](https://domduck.com/docs/ai-crawlers).