Privacy and what gets fetched

The AI Site Scanner runs on Toolz.dev's server, not in your browser. That is the reverse of most Toolz.dev tools, which run client-side: a browser is not allowed to read another site's robots.txt or well-known files, so the fetching has to happen on a server. From you, the scanner receives the URL you enter and your scan options. From the site, it requests one page, a fixed list of files at the site's root, and two DNS records, then stores the result as a report.

What do I send to Toolz.dev?

The URL, the options you picked (page or whole site, public or private, site type and checks), and, if you are signed in and ask for a private scan, your sign-in token so the server can confirm Pro. The scan endpoint checks your daily allowance with the same usage service as other Toolz.dev tools: it passes on your sign-in token if you have one, and the scanner page adds a device identifier; for a guest without one it uses a hash of your IP address. The command line and the MCP server send only the URL, visibility=public and, for a whole-site scan, mode=site.

Nothing about the site's content comes from your device. If the site is only reachable from your network, for example behind a VPN or on an intranet, the scanner cannot reach it either.

What does the scanner request from the site?

For every scan, from Toolz.dev's server:

Request	Time limit	Used by
The URL you entered, following up to 5 redirects	15 s	Every page check, HTTPS, status, time to first byte, headers
/robots.txt	15 s	robots-txt, ai-bot-rules, sitemap
/llms.txt	2.5 s	llms-txt
/llms-full.txt	2.5 s	llms-full-txt
/sitemap.xml	2.5 s	sitemap, whole-site page list
The page again, with Accept: text/markdown	2.5 s	markdown-negotiation (first 2,000 characters read)
/.well-known/api-catalog, /.well-known/oauth-authorization-server, /.well-known/oauth-protected-resource, /auth.md, /.well-known/mcp/server-card.json, /.well-known/agent.json, /.well-known/agent-skills.json, /.well-known/webmcp.json	2.5 s each	Protocol checks (first 400 characters read)
/.well-known/x402, /.well-known/mpp, /.well-known/ucp, /.well-known/acp	2.5 s each	Commerce checks
DNS TXT lookups for _ai.<domain> and _agents.<domain>		dns-aid

All of these are requested at every scan, even when a site-type filter skips the checks that use them. They are plain GET requests with no cookies and no login, the same as any crawler.

A whole-site scan then adds up to 2 child sitemaps (when /sitemap.xml is an index) and up to 5 pages from the sitemap, each with an 8 second limit and read up to 1.5 MB. Page scan vs whole-site scan explains which pages.

What user agent does the scanner use?

One, on every request: the URL, the root files, the Markdown request, the discovery paths, and a whole-site scan's child sitemaps and extra pages all send User-Agent: ToolzAIScan/1.0 (+https://toolz.dev/ai-scan). The link in it points at the scanner, so a site owner who sees it in their logs can find out what it is.

The scanner does not follow robots.txt rules for its own requests: it reads robots.txt to report what it says to AI crawlers, and it requests the other files whatever the rules say, because checking them is the point of the scan. It does not pretend to be GPTBot, ClaudeBot or any other AI crawler, so a site that treats those bots differently from other visitors may answer the scanner differently than it answers them. If your firewall or bot protection blocks the scanner, allow the user agent above; see Troubleshooting.

Which addresses will the scanner refuse to fetch?

Anything that is not on the public internet. The scanner fetches URLs supplied by anyone, so before it requests the URL, the root files or the discovery paths, it resolves the host name and refuses if any of its addresses is private. The extra pages of a whole-site scan are taken only from the same origin as the URL that passed this check, and each of those requests, including every redirect it follows, goes through the same check. This protects against server-side request forgery (SSRF), where a crafted URL makes a server fetch its own internal addresses. Refused:

Refused	Examples
Loopback and "this network"	127.0.0.0/8, 0.0.0.0/8, ::1, ::
Private networks (RFC 1918) and unique local IPv6	10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7
Link-local, including cloud metadata addresses	169.254.0.0/16, IPv6 addresses starting fe80
Carrier-grade NAT	100.64.0.0/10
192.0.x.x, multicast and reserved ranges	192.0.0.0/16, 224.0.0.0 and above, ff00::/8
Local host names	localhost, and names ending .localhost, .local or .internal
URLs with a username or password	https://user:[email protected]
Schemes other than http and https	ftp://, file://

The check is repeated for every redirect, so a public URL that redirects to a private address is stopped at that hop with "The site redirected to an address that is not publicly reachable." The connection is also pinned to the address that passed the check, so a DNS answer that changes between the check and the connection cannot slip through.

What is stored, and who can see it?

When a scan loads the site, the full result is stored on Toolz.dev as a report: the URL, domain, score, grade, site type, detected platform, visibility, time, and every check's outcome and detail line. It does not store the page's HTML or the files it fetched, only what the checks concluded about them, and the detail lines, which can quote short parts such as a Content-Signal header or the first characters of a well-known file.

	Public scan	Private scan (Pro)
Report stored with a link	Yes	Yes
Listed publicly	Yes, as the domain's newest public scan	No
Included in the totals on the scanner page	Yes	Yes

The browser's own Your scans list lives in local storage on your device and is never sent to Toolz.dev. Reports, sharing and history and Public and private scans, limits and Pro have the details.

Can I scan a site I do not own?

Yes. The scanner requests only public URLs, without logging in and without any testing beyond ordinary GET requests, as a search crawler would. If you are auditing someone else's site, keep in mind that a public scan puts that domain in the public directory; a private scan does not.