Does Disallow remove a page from Google?

No, and this is the single most common misunderstanding. Disallow stops Google from fetching the page; it does not remove the URL from the index. If other sites link to a blocked URL, Google can still list it - usually with no description, because it never read the page. To keep a page out of search results, allow crawling and add a noindex robots meta tag or an X-Robots-Tag header. If you block the URL, the crawler can never see the noindex you added.

Is robots.txt a way to hide private pages?

No. Robots.txt is a public file that anyone can read, and it is a request that well-behaved crawlers honour voluntarily - malicious scrapers ignore it entirely. Listing /internal-admin/ in your Disallow rules tells every attacker exactly where to look. Anything that must stay private needs authentication, an IP allowlist, or simply not being on the public internet.

How do Allow and Disallow interact when both match?

The most specific rule wins, where specificity means the length of the path pattern - this is the longest-match rule in RFC 9309. Given Disallow: /admin and Allow: /admin/public, the URL /admin/public is allowed because the Allow pattern is longer, while /admin/secret is blocked. If two patterns are exactly the same length, Allow wins. Order in the file does not matter, which surprises people coming from firewall rules.

What do the * and $ wildcards do?

An asterisk matches any run of characters, so Disallow: /*?sessionid= blocks any URL containing that query parameter. A dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?download=1, because the URL does not end at .pdf. Without the $, matching is a prefix match: Disallow: /admin blocks /admin, /admin/, and /administrator alike.

How do I block AI crawlers like GPTBot and ClaudeBot?

Add a group for them with Disallow: /. The Block AI crawlers preset puts GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider and anthropic-ai together in one group, one User-agent line each, while a separate `User-agent: *` group with Allow: / leaves Googlebot and Bingbot free to crawl. The Block button in the Known crawlers table adds a separate group for a single crawler instead. The preset replaces the whole form, so add your own rules afterwards. Note that Google-Extended only controls training use for Gemini and Vertex AI; it has no effect on Google Search rankings. Compliance is voluntary, so this stops cooperative crawlers, not determined scrapers.

Does Crawl-delay work?

It depends on the engine. Googlebot ignores Crawl-delay completely, and Google has retired its crawl rate setting in Search Console, so to slow Googlebot during an overload return 503 or 429 responses for a short time. Bing and several smaller crawlers honour it, treating the value as the minimum number of seconds between requests. Setting a large delay on a big site can mean pages never get crawled, so use small values, and fix the underlying server capacity if crawling is genuinely hurting you.

What happens if my robots.txt has a syntax error?

Crawlers silently discard lines they cannot parse and carry on with the rest, which is why a broken rule fails quietly instead of loudly. An unknown directive, a path missing its leading slash, or a rule placed above the first User-agent line simply does nothing, and you will not notice until traffic moves. The validator checks the file this tool generates, so a path missing its slash or a relative sitemap URL is reported with a line number. Lines that Parse into groups cannot place in a group, such as a rule above the first User-agent line, an unknown directive or a Noindex line, are dropped on import without a warning.

Should I list my sitemap in robots.txt?

Yes. A Sitemap: line with an absolute URL is the cheapest way to point every crawler at your URL list, and it works even for engines where you have never registered the site. You can list several sitemaps or a sitemap index. The line is global - it sits outside any user-agent group and applies to everyone regardless of where you place it in the file.

Is this robots.txt generator free to use?

Yes, it is free, no signup. The file is built, validated and tested in JavaScript inside your browser, and your paths, hostnames and rules are never uploaded, so you can draft the robots.txt for a staging environment safely. Copy and Download count toward the daily usage limit for the tool, and each one sends a usage request that says which action you took, not the file.

Can I validate my existing robots.txt file here?

Partly. Paste the file into Import an existing robots.txt and press Parse into groups: it becomes editable groups, and the validator then checks the rebuilt file for relative paths, empty rules, relative sitemap URLs, a blanket Disallow: / and duplicate crawlers. It does not fetch a live URL, and import drops comments, unknown directives and any rule above the first User-agent line without reporting them, so compare the output with the original. Line numbers refer to the generated file, not the one you pasted.

How do I test a URL against my robots.txt rules?

Type a path or full URL into Path or URL and a crawler token such as Googlebot into User-agent, and the result reads Allowed or Disallowed with the rule that decided it. With the example file, https://example.com/cart?x=1 as Googlebot is Disallowed by `Disallow: /cart`. Enter the token, not a full user-agent string: "Mozilla/5.0 (compatible; GPTBot/1.2)" matches no group by name and falls back to the `*` group. Only an exact token match counts, so Googlebot-Image does not inherit a Googlebot group.

Robots.txt Generator

Build, validate, and test a robots.txt file - user-agent groups, allow/disallow rules, sitemaps, AI crawler blocking, and a live URL tester

Presets:

User-agent group 1

Crawlers

Disallow

Allow

Crawl-delay (seconds, optional)

User-agent group 2

Crawlers

GPTBotClaudeBotCCBot

Disallow

Allow

Crawl-delay (seconds, optional)

Sitemap URLs

Host (optional, Yandex only)

Import an existing robots.txt

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /*?sessionid=
Allow: /admin/public/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Upload this to the root of your domain so it is served at /robots.txt.

Validation

0 errors · 1 warning

Line 10: `Disallow: /` blocks the whole site for "GPTBot, ClaudeBot, CCBot". That is intentional when refusing a specific crawler - just confirm it is the agent you meant.

Test a URL

Path or URL

User-agent

Allowed - `Allow: /admin/public/` in the "*" group is the longest matching rule for /admin/public/style.css, so it is crawlable.

The longest matching path pattern wins, and Allow beats Disallow on a tie - the same precedence real crawlers use.

Known crawlers

Click Block to add a group that refuses a crawler outright. Compliance is voluntary - this stops cooperative bots, not scrapers.

Token	What it is	Action
Googlebot	Google Search's main web crawler. Blocking it removes you from Google over time.
Googlebot-Image	Crawls images for Google Images. Block it to keep photos out of image search.
Bingbot	Microsoft Bing's crawler. Also feeds Bing-powered results in other products.
DuckDuckBot	DuckDuckGo's own crawler for its instant answers and index.
Baiduspider	Baidu's crawler, the dominant search engine in China.
YandexBot	Yandex's crawler, the leading search engine in Russia.
Slurp	Yahoo's legacy crawler. Still used for some Yahoo verticals.
Applebot	Apple's crawler, powering Siri and Spotlight Suggestions.
Twitterbot	Fetches link preview cards for X/Twitter. Blocking it kills your share previews.
facebookexternalhit	Fetches Open Graph previews for Facebook, Messenger, and WhatsApp.
GPTBotAI	OpenAI's crawler that collects public web content used to train its models.
ClaudeBotAI	Anthropic's crawler for gathering training data for Claude.
PerplexityBotAI	Perplexity's crawler, used to index pages its answer engine cites.
CCBotAI	Common Crawl's bot. Its public corpus is a common source of AI training data.
Google-ExtendedAI	Controls whether Google may use your content for Gemini and Vertex AI training. It does not affect Google Search ranking or indexing.
BytespiderAI	ByteDance's crawler, associated with TikTok and its AI products.
anthropic-aiAI	Legacy Anthropic token. Harmless to keep alongside ClaudeBot for older references.

Category:seo

Mode:Offline

Version:1.0.0

Access:Free

What is a Robots.txt Generator?

Robots.txt Generator interface on Toolz.dev - Build, validate, and test a robots.txt file - user-agent groups, allow/disallow rules, sitemaps, AI crawler blocking, and a live URL tester

A Robots.txt Generator builds the plain-text file that sits at the root of your domain and tells crawlers which parts of your site they may fetch. The format looks simple - a `User-agent:` line followed by `Allow:` and `Disallow:` rules - but the details bite: rules only apply inside a group, paths must begin with a slash, `*` and `$` behave in specific ways, and a single stray `Disallow: /` under `User-agent: *` can pull an entire site out of Google. This tool builds the file from a structured form so the syntax is correct by construction.

It does three jobs beyond generation. It lints the file it builds and flags the real footguns: relative paths, a bare `Disallow:` that silently means "allow everything", sitemap URLs that are not absolute, a very high Crawl-delay and blanket blocks, and notes when one crawler is split across two groups. It parses an existing robots.txt back into editable groups, so you can paste what is live today and fix it. And it tests a URL against your rules with the same longest-match precedence real crawlers use - the most specific matching pattern wins, Allow beats Disallow on ties, and groups naming the same crawler are combined - so you can prove a page is reachable before you deploy.

As a robots.txt file generator, the page opens on a worked example. The `User-agent: *` group disallows /admin/, /cart and /*?sessionid= and allows /admin/public/, and a second group refuses GPTBot, ClaudeBot and CCBot with `Disallow: /`. Test /admin/public/style.css as Googlebot and the result is Allowed, because `Allow: /admin/public/` is the longest matching rule. Change the path to /admin/settings and it flips to Disallowed by `Disallow: /admin/`. The validator shows one warning, confirming the `Disallow: /` is meant for those three crawlers. This free robots.txt generator needs no signup.

One thing worth stating plainly: robots.txt controls crawling, not indexing, and it is not a security boundary. A blocked URL can still appear in search results if other pages link to it, because Google can index a URL it has never fetched. To keep a page out of the index you need a `noindex` meta tag or an X-Robots-Tag header - which the crawler can only see if it is allowed to fetch the page. And the file is public: anyone can read yours at /robots.txt, so listing a secret directory there advertises it. Use authentication for anything private.

How to use the Robots.txt Generator?

Start from a preset or paste your current file

Pick a preset - Allow everything, Block everything, WordPress, Shopify, Next.js, or Block AI crawlers - to fill the form in one click, or paste your live robots.txt and let the tool parse it back into editable groups.

Build your user-agent groups

Add a group per crawler or set of crawlers, choosing from the known-bot list or typing a token. Give each group its Disallow and Allow paths, and a Crawl-delay if the engine you are targeting honours one.

Add sitemaps and read the validator

Add one or more absolute sitemap URLs. The validator runs on every keystroke and reports errors, warnings and notes, with a line number where one line is at fault. Fix anything red before you ship, and replace the example.com sitemap URL the presets put in.

Test a URL, then copy or download

Enter a path and a user-agent to see Allowed or Disallowed and exactly which rule decided it. When it behaves the way you expect, copy the file or download robots.txt and upload it to your domain root.

Key Features

Real Longest-Match URL Tester

Tests any path against any user-agent using RFC 9309 precedence - the longest matching pattern wins, Allow breaks ties - and names the exact rule that decided it

Built-in Validator

Catches relative paths, empty Disallow and Allow lines, non-absolute sitemap URLs, a Crawl-delay over 30 seconds, and a blanket Disallow: / that would drop your site from search

One-Click Presets

Allow everything, Block everything, WordPress, Shopify, Next.js, and Block AI crawlers each fill the whole form with a sensible, correct starting point

AI Crawler Controls

Block GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, and anthropic-ai in one click while keeping search engines fully welcome

Frequently Asked Questions

It must live at the root of the host it applies to, served at exactly /robots.txt - for example https://example.com/robots.txt. Crawlers do not look anywhere else. A file at /blog/robots.txt is ignored, and robots.txt on example.com does not govern shop.example.com or the http:// version if you serve one; each host and scheme needs its own file. Serve it as text/plain with a 200 status.

Related Tools

seo

Hreflang Generator

Free hreflang generator for multilingual and multi-region sites - build hreflang tags as HTML link tags, XML sitemap entries, or an HTTP header. Validates codes and x-default. Runs client-side.

HRG

seo

htaccess Redirect Generator

Generate Apache .htaccess 301 and 302 redirect rules from a list of old-to-new URLs, plus force-HTTPS and www canonicalization directives, all in the browser.

web

Domain Search

Check one name across dozens of TLDs at once. RDAP first, WHOIS fallback, and a verdict of available, taken or unknown, never a guess

marketing

Maps Scraper

Pull business listings for any search and place: name, address, category, rating, review count, phone and website, deduplicated and exportable as CSV.

Comments

0 comments

No comments yet. Be the first to share your thoughts!

robots.txt generatorrobots.txt file generatorfree robots.txt generatorrobots.txt generator freerobots.txt testerblock ai crawlersdisallow googlebotrobots txt validatorcrawl delayuser-agent directiveseo crawl controlblock gptbotsitemap directive

Command Palette