A structured benchmark of 409 publicly identifiable Australian business websites (commercial operating entities) across 10 industry groups, measuring which AI search crawlers can — and cannot — access them.
Of 314 practitioner domains whose crawler policy could be directly observed, 133 — 42.4% — block at least one crawler used by AI search systems to discover and cite content.
We separate observable AI-crawler policy from infrastructure non-response. A block declared in robots.txt is a policy decision; a 403, timeout or unscannable response is an access outcome, not evidence of crawler policy. These are reported separately below.
Of the blocked sites, 90.2% are broad access restrictions catching AI crawlers incidentally rather than AI-specific decisions.
Methodology update (v1.2). This report has been updated to align with the harmonised AI Crawler Access Study methodology used across Australia, the United States, Great Britain and Singapore. Version 1.2 applies a practitioners-only sample definition (commercial operating entities only), the standardised 21-user-agent crawler set, and revised handling of access-denied responses. Figures published under Version 1.0 are superseded. All results on this page reflect the Version 1.2 frozen dataset (n=314). Data: PTODA C01 Crawler v1.2 · series master frozen 17 June 2026.
Policy-layer figures are based on 314 practitioner domains whose robots.txt was successfully retrieved and parsed. A further 95 domains returned no observable policy (39 access-denied, 56 unscannable) and are reported separately in the Infrastructure layer section. They are not counted as open or blocked.
The central finding: Most blocked businesses are not actively choosing to exclude AI search. They have broad access restrictions set years ago that are now inadvertently catching AI crawlers. This is a configuration problem, not a strategic decision.
We separate observable AI-crawler policy from infrastructure non-response. A block in robots.txt is a policy decision. A 403, timeout or unscannable response is an access outcome, not evidence of crawler policy. These 95 domains are reported here and excluded from every policy-layer figure.
Why this matters: a domain that denies the crawler at the infrastructure layer has not expressed an AI-crawler policy — it has prevented one from being read. Treating such a response as “open” would overstate access; treating it as “blocked” would overstate restriction. Reporting it separately keeps the policy-layer figures based only on directly observed robots.txt behaviour.
Of the 133 sites blocking AI retrieval crawlers, the source of the block was classified into three categories.
The infrastructure-imposed subset is the most commercially significant finding: these site owners may be blocking AI search discovery without ever having made that decision. The indeterminate category — 51 Cloudflare-hosted sites — most likely represents explicit blocks, but the configuration path cannot be confirmed by automated means alone.
Block rates vary across the 10 sectors. Real Estate and Accounting & Finance are highest; Professional Services lowest. Rates are computed on policy-observed domains per sector (readable robots.txt only).
Real Estate at 57.1% and Accounting & Finance at 53.3% are the highest-blocking sectors. That matters, since AI search is reshaping how Australians find property and financial advice. Technology & SaaS at 50.0% remains among the study's most counterintuitive findings: the sector most aware of AI is among the most likely to be invisible to it. Professional Services at 23.3% is the most open.
Group A (retrieval/citation crawlers) drives the headline finding. Group B (training crawlers) is reported separately, because blocking training crawlers is often a deliberate and legitimate content-protection decision.
The Googlebot parity finding is the most important number in the dataset. Googlebot is blocked at 38.2%, right alongside the AI retrieval crawlers (GPTBot 40.8%, ClaudeBot 40.4%). This confirms that the vast majority of AI crawler blocks are broad restrictions, not targeted AI decisions. Businesses blocking AI crawlers are mostly also blocking Google. The 14 retrieval crawlers cluster tightly (37.9%–40.8%), indicating that where AI is blocked, it is typically blocked uniformly across operators rather than selectively.
Block rates by content management system among policy-observed domains. WordPress is the only platform with a large enough base for a reliable rate; the others are shown for completeness but rest on small samples and should be read with caution.
Most sites return no identifiable CMS signature, so platform-level rates are based on the minority that do. WordPress at 33.9% (n=56) tracks close to the overall sample average, the broadest cross-section of Australian business websites. Drupal at 10.0% (n=10) is directionally low, consistent with Drupal's enterprise and government skew where robots.txt is more deliberately managed, but the small base means this is indicative rather than conclusive. Webflow (n=4), Joomla (n=2) and Squarespace (n=1) have too few domains in this sample to report a meaningful rate.
Roles. This study is published by the Periodic Table of Digital Authority (PTODA), the methodology owner. It was conducted using the PTODA C01 Crawler v1.2, a deterministic robots.txt reference instrument, under PTODA C01 Crawler Methodology v1.2. AUTHORITY44 provided technical infrastructure and execution support as commercial operator. Douglas Lord is the founder of both PTODA and AUTHORITY44; this relationship is disclosed in full. The sample was constructed from named public directories with no reference to commercial relationships. The methodology is fully documented and reproducible. This study publishes aggregate, anonymised findings only. No named individual site results are published.
Attribution chain: Douglas Lord (researcher, author) · Periodic Table of Digital Authority (publisher & methodology owner) · PTODA C01 Crawler v1.2 (research instrument) · AUTHORITY44™ (commercial operator) · Digital Dominator Pty Ltd ABN 28 616 931 116 (operating entity).
Intellectual property notice: This study, its methodology, findings, data, and all associated content are the original work of Douglas Lord and the property of Digital Dominator Pty Ltd (ABN 28 616 931 116). The Periodic Table of Digital Authority™ is a coined framework and trade mark pending (TM 2644497). AUTHORITY44™ is a trade mark pending (TM 2643932). All rights reserved.
You may cite findings from this study with appropriate attribution identifying the author (Douglas Lord), the publisher (Periodic Table of Digital Authority — periodictableofdigitalauthority.com), and the research instrument (PTODA C01 Crawler v1.2). You may not reproduce this study in full, present these findings as your own research, or use the framework name or trade marks without prior written consent. Use of this research is subject to the Terms of Use.