Data sources
Every source Domduck reads, what its numbers mean, which direction is better, and how often it is re-read.
Plain markdown: /docs/sources.md
Every source Domduck reads, and what its numbers mean.
| Source | Reads | Cadence |
|---|---|---|
ahrefs_dr | Domain Rating. 0-100, higher is better. | daily |
tranco | Tranco rank. top 1M, lower is better. | daily |
openpagerank | OpenPageRank. 0-10, higher is better. | daily |
cloudflare_radar | Cloudflare Radar. rank plus content categories. | daily |
rdap | Registration. registration date and status. | once |
self_tls | TLS certificate. issuer, validity, days remaining. | daily |
self_dns | DNS. A, AAAA, MX, NS, SPF, DMARC, DNSSEC. | daily |
robots_txt | AI crawler policy. count of 8 tracked AI crawlers blocked, lower is better. | daily |
Direction is not obvious, so it is stated
tranco, cloudflare_radar and robots_txt are better when lower. ahrefs_dr and openpagerank are better when higher. Every history the MCP server returns carries a higherIsBetter field for this reason: a rank series read by something that assumes bigger is better is a chart drawn upside down.
Every reading is a dated observation
Nothing is ever overwritten. A reading is appended with the timestamp it was taken at, and the current value is a query over that history rather than a column.
The consequence worth knowing before you read a series: a missing day means nobody looked. It does not mean the value was zero and it does not mean the value held. That distinction is why the charts draw a gap for a day with no reading and a dashed line for a value carried past the last reading, and it is why null appears in an API response where a lazier design would send 0.
Notes per source
- **
ahrefs_dr** Domain Rating, 0-100, on a logarithmic scale, so 80 to 81 is a much larger move than 20 to 21. - **
tranco** A research list that averages 30 days of several popularity rankings, so it is far more stable than any single day's list. Only the top 1M exists; outside it, there is correctly no reading rather than a bad one. - **
openpagerank** 0-10. Free, coarse, and useful mainly as a second opinion when Domain Rating moves. - **
cloudflare_radar** An exact rank for roughly the top 100 domains and a bucket ceiling below that, plus content categories. - **
rdap** Registration date, registrar and status, from the registry through IANA's bootstrap. Read once: it does not change. - **
self_tls** Our own TLS handshake with the domain. Issuer, validity window and days remaining. - **
self_dns** A, AAAA, MX, NS, SPF, DMARC and DNSSEC, through a fixed DNS over HTTPS resolver. - **
robots_txt** The AI crawler policy, plus anllms.txtpresence probe on the same pass.
The eight AI crawlers
| Agent | Operator | Blocking it means |
|---|---|---|
GPTBot | OpenAI | No ChatGPT browsing or training |
OAI-SearchBot | OpenAI | No ChatGPT search surface |
ClaudeBot | Anthropic | No Claude retrieval |
PerplexityBot | Perplexity | No Perplexity citation |
Google-Extended | Out of Gemini and AI Overviews grounding, Search ranking unaffected | |
Applebot-Extended | Apple | Out of Apple Intelligence training |
CCBot | Common Crawl | Out of most open training corpora |
Bytespider | ByteDance | Out of TikTok and Doubao |
Full detail on the AI crawler page.