AI Site Scanner docs
How the Toolz.dev AI Site Scanner grades a website for AI agents, what it checks, how the score works, and where to find the CLI, MCP server and API.
The AI Site Scanner grades how well a website works for AI agents such as ChatGPT, Claude, Perplexity and Google's AI features. You give it a URL, it fetches the page and the files an agent looks for, runs 31 weighted checks in five categories, and returns a score from 0 to 100, a letter grade from A+ to F, and a fix prompt for every failed check that you can paste into an AI coding assistant.
These docs are the reference for it: every check and its weight, how the score and grade are computed, what gets stored and who can see it, and how to run the same scan from a terminal, a CI job, an AI assistant or your own code. If you only want a grade, the Quickstart takes about a minute.
What does the AI Site Scanner check?
It checks the signals an AI agent relies on when it reads a site, which are not the same signals a search ranking audit looks at. The 31 checks fall into five categories:
| Category | The question it answers | Checks |
|---|---|---|
| Discoverability | Can agents find the site and its map of content? | 7 |
| Content Accessibility | Is the substance readable by machines without a browser? | 9 |
| Bot Access Control | Does the site tell AI crawlers what they may do? | 3 |
| Protocol Discovery | Machine-readable entry points for agent protocols | 8 |
| Commerce | Emerging agentic payment and checkout protocols | 4 |
That covers robots.txt and its rules for crawlers like GPTBot and ClaudeBot, the XML sitemap, llms.txt, schema.org JSON-LD, whether the text is in the server-rendered HTML, and a set of /.well-known/ files for agent protocols. What the scan checks lists all 31 with their weights and exactly what makes each one pass.
Why does agent-readiness matter?
An AI assistant that answers a question from the web has to fetch your page, read it and decide whether to cite it, usually without running JavaScript and often in a few seconds. A page that blocks the crawler in robots.txt, renders its text only in the browser, or carries no structured data is hard for that assistant to use, even if it ranks well in a search engine. The scanner measures those failure points one by one, so you can see which ones apply to your site and fix them in order of weight.
How does it work, in one paragraph?
Toolz.dev's server fetches the URL you enter, follows up to five redirects, and in parallel requests /robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml, a Markdown version of the page, twelve discovery paths such as /.well-known/agent.json and two DNS TXT names. Each check reads those responses and returns pass, fail or not applicable. Every check has a weight from 2 to 8; a category's score is the share of its weight that passed, and the overall score is the weighted average of the categories. Unlike most Toolz.dev tools, which run client-side, the scanner runs on the server, because a browser cannot fetch another site's files. Privacy and what gets fetched has the details.
Where do I find what?
| If you want to | Read |
|---|---|
| Scan your first site | Quickstart |
| Grade more than the homepage | Page scan vs whole-site scan |
| See every check, its weight and what passes | What the scan checks |
| Understand the score, the grade and the site-type filter | Scores, grades and scan filters |
| Turn failed checks into changes | Fixing issues with fix prompts |
| Share a report or see a domain's past scans | Reports, sharing and history |
| Keep a scan out of the public list, or check your limits | Public and private scans, limits and Pro |
| Scan from a terminal or fail a CI build | Command line (npm package) |
| Scan from Claude, Codex, Gemini CLI or Cursor | MCP server |
| Call the scanner from your own code | HTTP endpoint |
| Know what the scanner requests and stores | Privacy and what gets fetched |
| Fix a scan that failed or was refused | Troubleshooting |
What does it cost?
Scanning is free, no signup, within a daily allowance per account type that the scanner page shows under its controls. A free account gets a larger allowance than a guest, and Toolz Pro makes it unlimited and adds private scans. Scans are public by default: the report gets a link, and the newest public scan of each domain is listed in the public scan directory.
Several Toolz.dev tools help with the fixes themselves: the robots.txt generator, the schema markup generator, the sitemap generator and the meta tag generator each produce a file or tag that one of the checks looks for.