The deterministic robots.txt access scanner used for the Global Digital Authority Benchmark Series crawler-access studies. This document is the versioned, reproducible specification studies cite. v1.5 is the publication baseline for the complete C01 series under Governance Decision Record GDR-001.
The C01 Crawler determines, for a defined sample of domains, the publicly declared access configuration for major AI crawler user-agents — by retrieving and parsing each domain's robots.txt, homepage robots meta tags, and relevant HTTP headers, then detecting CMS and CDN/host signals.
It measures declared crawler-access policy, not crawler behaviour and not citation outcomes. It is deterministic: the same domain in the same state returns the same classification, which makes every study reproducible by re-running the instrument against the published sample.
v1.2 separates observable crawler policy from infrastructure non-response. The two are never mixed, and policy rates are computed only on domains whose robots.txt could actually be read.
A block declared in robots.txt is a policy decision. A 403, timeout or unscannable response is an access outcome, not evidence of crawler policy. Reporting the two layers separately prevents over-counting blocks and over-counting openness — results are reported only where behaviour was directly observed.
A single binary blocked/not-blocked measure combines materially different forms of robots.txt restriction into one prevalence figure. From v1.4 the instrument captures every disallow path in the governing group and classifies each domain into one of five graded access classes. Under GDR-001, these graded measures are the publication baseline for the whole C01 series; the earlier binary measure is superseded for publication purposes.
Path classification. Every disallow path in the governing robots.txt group is classified against a frozen pattern library into one of three path classes:
Conservative defaults. Ambiguous paths (e.g. a bare /shop or /services) and paths matching no library pattern are classified secondary — the restrictive direction — and logged per domain in dedicated columns for review before publication. A carve-out classifies data/API endpoints beneath primary directories (e.g. a provider directory's …/data feed) as functional, so plumbing beneath primary content is never counted as a primary restriction. Locale-prefixed paths (/de/…, /en-au/…) are classified both raw and locale-stripped, taking the more restrictive known result — locale normalisation can resolve an unknown path into a known class but can never make a path look more open than a genuine match.
Domain classification. Each policy-observed domain is then assigned one of five access classes by strict precedence — whole-site > primary > secondary > functionally open > fully open:
Disallow: / or /* with no overriding Allow: /.Reported restriction rates. Three graded rates are computed on the policy-observed denominator, in increasing breadth:
Why the binary measure is superseded. The binary heuristic counted functional-infrastructure disallows — carts, search endpoints, APIs, asset bundles — as "blocking," inflating the apparent restriction rate. For the United States frozen 808-domain dataset, the v1.2 binary headline of 42.2% resolves under the graded taxonomy into 3.8% whole-site exclusion and 10.7% meaningful restriction, with the difference being functional and secondary disallows the binary measure over-counted. Superseded v1.2 publications are retained in the research record with an explicit notice, per GDR-001.
v1.3 adds a machine-readable-guidance layer to the v1.2 access instrument. On each domain, in the same scan pass, the crawler assesses whether the site has adopted llms.txt. The Layer 1 access methodology is unchanged and reproducible; this is an additive layer, not a revision to access measurement.
The crawler fetches /llms.txt and, immediately before it, a randomly-named control path that cannot exist. A file is recorded as present only when /llms.txt returns a genuine HTTP 200 and the control path does not return matching content — this excludes servers that return 200 for every path (soft-404s) from being miscounted as adopters. It is the single most important control in the adoption measurement.
Presence is separated from conformance. A present file is recorded as conformant only when it is non-empty, is not HTML (by body or content-type), is markdown-structured with at least one heading, and is not a placeholder. Adoption and conformance are reported as distinct rates, so a present-but-invalid file (empty, HTML, or placeholder) counts toward adoption awareness but not toward valid guidance.
Classification: present_conformant · present_nonconformant · absent · unscannable. As with the access layer, infrastructure non-response (timeout / 5xx) is excluded from adoption denominators. The paired analysis — llms.txt presence against AI-crawler access — uses only domains where both layers were observable. For every present file, the crawler also records byte size, markdown link count, content-type, and a SHA-256 content hash, retained to support later analysis of whether richer files correlate with AI citation.
v1.5 adds operational run machinery only. Classification logic is untouched: the same domain in the same state produces the same result under v1.4 and v1.5. What changes is how a batch run treats domains the instrument could not observe.
Bounded recovery. A batch runs the full sample once, then re-crawls only unresolved domains — those classified access-denied or unscannable — up to three attempts in total. Each attempt runs in a fresh randomised order under a logged, deterministic seed; each attempt's raw output is written to its own auditable file; resolved rows from later attempts replace the unresolved rows for the same domain, merged without duplicates. Domains still unresolved after the final attempt are frozen as final infrastructure failures, and the run reports initial observed, recovered, and final unresolved counts. A retry is a recovery of an observation, never a change to what a domain's result is — and an interrupted run aborts the recovery loop entirely rather than silently producing a falsely complete merged dataset.
Failure decomposition. Every failed measurement-critical fetch records a concrete failure mode — dns, connect, timeout, reset, tls, redirect_loop, or http_401/403/429/5xx — with per-attempt failure history retained on recovery-merged rows. The infrastructure layer is thereby reported as decomposed causes rather than an undifferentiated bucket.
The instrument is versioned. Each version is dated; the graded access taxonomy is frozen across the v1.5 series.
v1.2 — Access benchmark. robots.txt access measurement; the policy / infrastructure two-layer reporting model; access-denied vs unscannable separation. Instrument for the superseded binary-measure AU / US / GB / SG access studies, retained in the research record under GDR-001 notices.
v1.3 — Added llms.txt assessment layer. Adds llms.txt presence and conformance (soft-404 control, capture-now fields) in the same scan pass. No change to access-layer methodology — v1.3 is a strict superset of v1.2. Instrument for the AI Accessibility & Machine-Readable Guidance Benchmark.
v1.4 — Graded access taxonomy. Captures the complete disallow set of the governing group per domain and introduces the five-class access taxonomy with the strict / meaningful / expanded restriction rates, conservative ambiguous/unknown-path defaults with per-domain logging, and the locale-normalisation rule.
v1.5 — Publication baseline (current, GDR-001). Adds the bounded recovery run protocol and failure-mode decomposition; classification unchanged from v1.4. Adopted 27 June 2026 as the publication baseline for the complete C01 series. Frozen instrument SHA-256 a8a17d5d…3a5e55.
Two groups, interpreted differently. The crawler list is frozen across all volumes so that findings are comparable across countries and dates. New crawlers are not added mid-series.
Googlebot is included as a baseline. If a site blocks Googlebot at the same rate it blocks AI crawlers, the block is a broad restriction rather than an AI-specific decision — the single most important interpretive control in the method.
On Grok (xAI): deliberately excluded. xAI publishes no crawler documentation and its retrieval traffic uses residential-IP rotation with spoofed browser user-agents, presenting no declared user-agent and honouring no robots.txt contract. It cannot be measured by a robots.txt instrument, and including a token for it would produce meaningless results. Microsoft Copilot is covered by Bingbot; Gemini retrieval by Googlebot and Google-Extended.
The primary failure mode is misclassifying benign housekeeping directives as AI blocks. The following are never counted as blocks, in any volume:
Disallow: /wp-admin/ — standard admin housekeeping. Not an AI block.Crawl-delay: N — a politeness directive at any value. Recorded, never counted as blocking.Sitemap: declarations — informational, never a block.Disallow: — explicitly means allow all./cart, /search, /admin/, /login) — housekeeping, not content/AI blocks.A user-agent is classified blocked only where the governing robots.txt group disallows / or the homepage/primary content path. The classifier is validated against fixture tests before any batch is run.
https://{domain}/robots.txt; record HTTP status. 404 = no robots = open by default. 401 / 403 / 429 = access denied (infrastructure layer). Timeout / 5xx / connection failure = unscannable.*); determine whether it path-blocks the homepage/root for that bot.<meta> (noindex/nofollow/noai) and X-Robots-Tag headers. Recorded as secondary signals; robots.txt is primary.Samples must be named, public, reproducible and unbiased — never "sites we happened to scan."
a8a17d5d…3a5e55; validated on the US 808-domain frozen dataset.Future versions will be published at distinct URLs (e.g. /methodology/c01-crawler-v2) so each study cites a fixed, immutable specification.
Periodic Table of Digital Authority (PTODA) owns and maintains this methodology. The PTODA C01 Crawler is the reference instrument. Shared ownership is stated openly — the methodology's credibility rests on versioning, traceable datasets, disclosed limitations, and claims proportionate to evidence, all of which are public.
Cite as: PTODA C01 Crawler Methodology v1.5 (Periodic Table of Digital Authority, 2026), ptoda.org/methodology/c01-crawler-v1. Frozen instrument SHA-256: a8a17d5deb721a0653d2841fc3339efd2982ec3a5d26951a5987c8238a3a5e55. The Periodic Table of Digital Authority™ (TM 2644497) and AUTHORITY44™ (TM 2643932) are trade marks pending. © Digital Dominator Pty Ltd ABN 28 616 931 116.