What the scan checks
All 31 checks the AI Site Scanner runs, in five categories, with each check's weight, the site types it applies to and exactly what makes it pass.
The AI Site Scanner runs 31 checks in five categories. Each check has a weight from 2 to 8 that sets how much it counts toward the score, applies to content sites, API and application sites or both, and ends as pass, fail or not applicable. This page lists every check with its id (the name the HTTP endpoint and the checks filter use) and the exact condition it tests.
"All" in the Applies to column means the check runs for both site types. "Homepage" in the tables means the URL you scanned, which does not have to be the homepage.
Discoverability: can agents find the site?
Can agents find the site and its map of content? Seven checks, total weight 32.
| Check | Id | Weight | Applies to | Passes when |
|---|---|---|---|---|
| robots.txt present & valid | robots-txt |
6 | All | /robots.txt answers 2xx and contains at least one User-agent group |
| XML sitemap declared & reachable | sitemap |
5 | All | /sitemap.xml answers 2xx, or robots.txt has a Sitemap: line with an http or https URL |
| Link: discovery headers | link-headers |
3 | All | The page's HTTP Link header includes a rel of service-desc, service-doc, api-catalog, alternate, help, documentation or describedby |
| DNS-AID records | dns-aid |
2 | All | A DNS TXT record exists at _ai.<domain> or _agents.<domain> |
| Served over HTTPS | https |
7 | All | The final URL, after redirects, is https:// |
| Healthy HTTP status | http-status |
5 | All | The final response status is 200 to 299 |
| Fast time-to-first-byte | ttfb |
4 | All | The response headers arrive in under 800 ms, measured from Toolz.dev's server, redirects included |
robots.txt follows the Robots Exclusion Protocol, RFC 9309. Link headers are defined in RFC 8288. The DNS-AID check is marked not applicable, rather than failed, when the DNS lookup itself cannot be completed.
Content Accessibility: can machines read it?
Is the substance readable by machines without a browser? Nine checks, total weight 48. This is the heaviest category.
| Check | Id | Weight | Applies to | Passes when |
|---|---|---|---|---|
| Markdown content negotiation | markdown-negotiation |
4 | Content | Requesting the page with Accept: text/markdown returns a Content-Type containing markdown, or text/plain that looks like Markdown rather than HTML |
| llms.txt present | llms-txt |
8 | Content | /llms.txt answers 2xx, contains a # heading line and is longer than 40 characters |
| llms-full.txt (full content) | llms-full-txt |
3 | Content | /llms-full.txt answers 2xx with a non-empty body |
| JSON-LD structured data | json-ld |
7 | Content | The server-rendered HTML has at least one <script type="application/ld+json"> block and every block parses as JSON |
| Semantic HTML structure | semantic-html |
5 | Content | Exactly one <h1>, and at least three of <main>, <header>, <nav>, <footer>, <article> |
| Server-rendered content | ssr-content |
8 | Content | The <body> text of the raw HTML, before any JavaScript runs, is at least 300 words |
| Title, description & OpenGraph | meta-og |
5 | Content | A <title>, a <meta name="description"> and at least three og: meta tags |
| Canonical URL | canonical |
2 | Content | A <link rel="canonical"> with an href |
| No noindex / noai traps | noindex-guard |
6 | All | <meta name="robots"> does not contain noindex, noai or noimageai |
Content negotiation with the Accept header is defined in RFC 9110. The llms.txt format is described at llmstxt.org; the scanner checks only for a heading and a minimum length, not the full structure. Structured data vocabularies are at schema.org, and the landmark elements are defined in the WHATWG HTML standard.
Two details worth knowing. The Markdown check is not applicable when the second request gets no response. And the JSON-LD check fails with a specific message when JSON-LD appears only inside a JavaScript payload, because a crawler that does not run scripts never sees it.
Bot Access Control: are the rules clear?
Does the site tell AI crawlers what they may do? Three checks, total weight 13.
| Check | Id | Weight | Applies to | Passes when |
|---|---|---|---|---|
| AI bot rules in robots.txt | ai-bot-rules |
8 | All | robots.txt exists, none of the ten AI crawlers below is disallowed from /, and the User-agent: * group does not disallow / |
| Content Signals (AI-use preferences) | content-signals |
3 | All | The response has a Content-Signal header, or the page has <meta name="content-signal"> |
| Web Bot Auth (signed agents) | web-bot-auth |
2 | All | The page's response includes an Accept-Signature header (RFC 9421 section 5.1), which invites agents to sign their requests |
The ten crawlers ai-bot-rules tests, each matched to its most specific User-agent group as RFC 9309 describes:
| Crawler | Operator |
|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI |
| ClaudeBot | Anthropic |
| PerplexityBot | Perplexity |
| Google-Extended | |
| CCBot | Common Crawl |
| Applebot-Extended | Apple |
| Amazonbot | Amazon |
| meta-externalagent | Meta |
A site with no AI-specific groups still passes, as long as the wildcard group lets everyone in. The detail line then says the bots are "allowed by the default wildcard policy". If you want to generate the groups rather than write them, the robots.txt generator does it.
Protocol Discovery: are there agent entry points?
Machine-readable entry points for agent protocols. Eight checks, total weight 22, all for API and application sites. Each one requests one path with a 2.5 second limit and passes when that path answers 2xx with the file itself: a body that is not an HTML page, and for the JSON files one that starts with { or [. A site that answers every path with its HTML shell (a catch-all route or soft 404) fails these checks, because an agent would get the same page.
| Check | Id | Weight | Path requested |
|---|---|---|---|
| API catalog (/.well-known/api-catalog) | api-catalog |
4 | /.well-known/api-catalog (RFC 9727) |
| OAuth authorization-server metadata | oauth-auth-server |
3 | /.well-known/oauth-authorization-server (RFC 8414) |
| OAuth protected-resource metadata | oauth-protected-resource |
3 | /.well-known/oauth-protected-resource (RFC 9728) |
| auth.md (agent auth guide) | auth-md |
2 | /auth.md |
| MCP server card | mcp-card |
3 | /.well-known/mcp/server-card.json |
| A2A agent card | a2a-card |
3 | /.well-known/agent.json |
| Agent Skills manifest | agent-skills |
2 | /.well-known/agent-skills.json |
| WebMCP declaration | webmcp |
2 | /.well-known/webmcp.json |
Commerce: can agents pay?
Emerging agentic payment and checkout protocols. Four checks, weight 2 each, total 8, all for API and application sites. Each passes when its path answers 402 Payment Required, or 2xx with a body that is not an HTML page.
| Check | Id | Weight | Path requested |
|---|---|---|---|
| x402 payment discovery | x402 |
2 | /.well-known/x402 |
| MPP (Machine Payments Protocol) | mpp |
2 | /.well-known/mpp |
| UCP (Universal Commerce Protocol) | ucp |
2 | /.well-known/ucp |
| ACP (Agentic Commerce Protocol) | acp |
2 | /.well-known/acp |
The x402 check also passes when the page's response carries a header whose name contains x402, x-payment or payment-required.
When is a check "not applicable"?
A check that cannot be evaluated is left out of the score instead of failing. That happens in three cases: the DNS-AID lookup could not complete, the Markdown re-request got no response, or a check hit an unexpected error while running (its detail line then starts "Check could not be evaluated"). A category in which no check could be evaluated is left out of the report entirely. Scores, grades and scan filters shows how that feeds into the numbers.
Where do the weights come from?
They are set in the scanner's check registry and are the same for every site. The heaviest checks (weight 7 or 8) are the ones that decide whether an agent can use the page at all: HTTPS, llms.txt, JSON-LD, server-rendered text and the AI crawler rules. The lightest (weight 2) cover protocols that very few sites publish yet, such as DNS-AID and the commerce files, so a missing one costs little.