How Domduck reads a domain
Domduck writes one dated row per check and never overwrites one, so a gap in a chart means nobody looked rather than nothing happened.
Domduck writes one dated row per check and never overwrites one. That single decision produces every other rule on this page, and it is the reason a gap in a chart means nobody looked rather than nothing happened.
One row per check
Every reading from every source is stored as its own row with its own timestamp. Nothing is updated in place, nothing is compacted, and a value that did not change still gets a row.
That last part is the one that costs storage and buys correctness. If rows were written only when a value changed, a missing day would mean "it held", and a day nobody read would look identical to it. The two facts would be permanently merged and no later analysis could separate them.
What a chart is actually saying
Three different things, drawn three different ways:
| Drawing | Meaning |
|---|---|
| Solid line through a point | A real reading, on that day, from that source |
| Gap | No reading. Nobody looked, or the fetch failed. |
| Dashed continuation | The last known value, carried forward past the final reading so the line reaches today |
A carried-forward value is a drawing decision, not a measurement, and it is drawn differently for exactly that reason. Structured data published by Domduck uses the real readings only, never the carried-forward ones.
Zero is never used for a missing day. Zero is a score a domain can hold, and on a rank axis where lower is better, a zero would be the best result ever recorded.
Cadence, per source
| Source | How often | Why |
|---|---|---|
| Domain Rating | daily | The endpoint publishes today's value and no history |
| Popularity rank | daily, from the whole list | One file gives every tracked domain a rank the same day |
| AI crawler policy | daily for tracked domains, slower across the corpus | The file has no history, so the reading is the record |
| Certificate | daily | Days remaining changes daily, and so does the answer to "what is served now" |
| DNS | daily | A record change is an event worth dating |
| Registration | once | A registration date does not change |
The AI crawler pass sends the previous reading's ETag. Most files are unchanged, so the answer is a 304 and the prior verdicts carry forward with a fresh timestamp. A conditional request that returns 304 is still a reading: somebody looked, and the answer was the same.
Three states, not two
Allowed by name, blocked, and never mentioned. Merging the third into the second produces a number that cannot be compared with itself later, because a rise could be adoption or could be silence.
What Domduck does not do
- It does not backfill. History starts when a domain is added, because the sources publish today rather than the past. Rank is the one exception, and it arrives with a short window.
- It does not infer intent. A record says GPTBot was blocked on 12 March. It never says the site decided to block AI.
- It does not claim enforcement. robots.txt is a declaration. Whether a crawler obeyed it is a different measurement, from server logs, and Domduck does not have your server logs.
- It does not delete on downgrade. There is no downgrade, and a series that can be deleted is not a record.
Where to check the work
/status reports whether each source collected recently, measured from the readings themselves rather than from whether the page loads. A source that is built but not running says so. A source with no adapter says that too.
/docs/sources lists every source, its scale, which direction is better, and how often it is read.
Related
- AI crawlersAI crawlers are the agents that fetch web pages to train models or to cite them in answers. Eight of them matter, and they are eight separate decisions.
- Domain RatingDomain Rating is Ahrefs’ 0 to 100 score for the strength of a domain’s backlink profile. It is logarithmic, relative, and it is not a Google metric.
Primary source: The data sources, in full