The most expensive four characters I have ever seen shipped were Disallow: /. A team pushed a staging robots.txt to production during a release, nobody noticed, and over the following ten days Google quietly dropped most of the site's pages. Traffic did not crash in a way that set off alarms; it drained. By the time somebody thought to check /robots.txt, the recovery was a month of re-crawling. The file that caused it was four lines long.
Robots.txt is deceptively simple. It is plain text, it has maybe six directives worth knowing, and the syntax fits on an index card. That simplicity is exactly why it goes wrong: there is no build step, no linter in your CI, no type system, and no error message. If you write a rule that a crawler cannot parse, the crawler silently ignores that line and carries on. Your file looks fine, your site looks fine, and something you thought was blocked is being crawled — or something you needed crawled is not.
I build toolz.dev and I write robots.txt files for staging environments, client sites, and my own projects constantly, so I built the Robots.txt Generator to make the failure modes visible. It builds the file from a structured form, so the syntax is right by construction. It lints what you have written and calls out the footguns with line numbers. And it tests a URL against your rules using the same longest-match precedence real crawlers use, so you can prove a page is reachable before you deploy rather than after Search Console tells you it is not.
TL;DR: Robots.txt tells crawlers what they may fetch. It does not remove pages from search results, and it does not protect anything private — the file is public, and compliance is voluntary. Rules only apply inside a
User-agent:group, paths must start with/,*matches anything, a trailing$anchors the end, and when two rules match, the longest pattern wins with Allow breaking ties. The Robots.txt Generator builds, lints, and tests all of that in your browser.
What robots.txt actually controls
Robots.txt is a crawl directive, not an index directive. That single distinction explains most of the confusion around it.
When a crawler wants to fetch a URL on your host, it first fetches https://yourhost/robots.txt and checks whether the URL is permitted. If a Disallow rule matches, a well-behaved crawler does not request the page. That is the entire mechanism. It governs the HTTP request, and nothing else.
What it does not do is remove a URL from the search index. Google can and does index URLs it has never fetched, using only the anchor text and context of links pointing at them. If you block /pricing in robots.txt and fifty sites link to /pricing, Google may still show that URL in results — typically with no description, because it never read the page. Blocking a URL you want out of search actively works against you: the noindex tag that would remove it lives inside the page, and a crawler that is not allowed to fetch the page can never see it. The correct sequence is to allow crawling, serve <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header, wait for the page to drop out, and only then block it if you want to save crawl budget.
It also does not protect anything. Robots.txt is served publicly at a well-known path; anyone, including every attacker on the internet, can read yours in a browser. A Disallow: /internal-admin-v2/ line is a signpost pointing at the thing you wanted hidden. Malicious scrapers ignore the file entirely — honouring it is a voluntary convention, not an enforcement mechanism. Anything that must be private needs authentication or an IP allowlist. Robots.txt is a request, politely made, to crawlers who choose to listen.
Key features of the Robots.txt Generator
Structured group editor
The file's grammar is groups: one or more User-agent: lines followed by the rules that apply to them. Writing that by hand invites the single most common structural bug, which is a rule floating above the first User-agent: line where it applies to nobody. The generator builds groups as first-class objects, so every rule you add necessarily belongs to a group.
A validator that reports the real footguns
The linter runs on every keystroke and reports errors, warnings, and notes with line numbers. It flags rules outside any group, paths missing their leading slash, a bare Disallow: with no value (which means "allow everything" and is almost never what the author meant), sitemap URLs that are not absolute, $ used anywhere but the end of a pattern, unknown directives, duplicate user-agent groups, and — loudly — a Disallow: / sitting under User-agent: *.
A real URL tester
Enter a path and a user-agent and the tool reports Allowed or Disallowed, plus the exact rule that decided it. The precedence logic is the real one from RFC 9309: the longest matching path pattern wins regardless of the order rules appear in the file, and if two patterns are the same length, Allow beats Disallow. This is the part people most often get wrong from intuition, because it does not behave like a firewall's first-match-wins rules.
One-click presets
Allow everything, Block everything, WordPress, Shopify, Next.js, and Block AI crawlers each fill the entire form with a correct, sensible starting point. The WordPress preset blocks /wp-admin/ while carving out /wp-admin/admin-ajax.php, which many hand-written files forget and thereby break front-end functionality that Google needs to render the page.
AI crawler controls
One click adds refusal groups for GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, and anthropic-ai, while leaving Googlebot and Bingbot free to crawl. The known-crawler table explains what each bot actually does, so you are making an informed choice rather than pasting a list from a blog post.
Parses your existing file
Paste the robots.txt you already serve and the tool reads it back into editable groups. Most robots.txt work is not greenfield — it is auditing and repairing something written three years ago by someone who has left — and starting from what is actually live is the only sane way to do that.
Everything stays in your browser
The file is generated, linted, and tested entirely in client-side JavaScript. Nothing is uploaded, so you can safely draft rules for a staging host or an unannounced section of a site, and the tool works offline once loaded.
How to use the Robots.txt Generator
Step 1: Start from a preset, or import what you already have
If you are starting fresh, pick the preset closest to your stack. If you already serve a robots.txt, fetch it from https://yoursite.com/robots.txt, paste it into the import box, and click Parse into groups. The tool reconstructs the group structure, and the validator immediately tells you what is wrong with the file you have been running in production.
Step 2: Build your user-agent groups
Each group targets one or more crawlers. Add crawlers from the known-bot list or type a token directly — tokens are matched case-insensitively, so googlebot and Googlebot are equivalent. A group with * is the catch-all: it applies to any crawler that does not have a more specific group of its own.
The critical thing to internalise here is that crawlers obey exactly one group. Googlebot reads the Googlebot group if one exists and ignores the * group entirely — it does not merge them. If you create a Googlebot group to add one rule, you must repeat every rule from the * group inside it, or Googlebot will silently stop obeying all of them. This surprises almost everyone the first time.
Step 3: Add rules, sitemaps, and read the validator
Add Disallow paths for what you want crawlers to skip, and Allow paths for the carve-outs inside them. Add a Crawl-delay only if you are targeting an engine that honours one. Add your sitemap URLs — these must be absolute, including scheme and host, and they sit outside any group, applying to everyone.
Then read the validator. Errors are things that will not work. Warnings are things that probably do not mean what you think. Notes are opportunities. Fix everything red before you go further.
Step 4: Test the URLs you actually care about
This is the step people skip and should not. Take the three or four paths whose crawl status genuinely matters — your product pages, your blog, the admin route you think you blocked, the CSS file Google needs to render your layout — and test each one against Googlebot. The tool tells you Allowed or Disallowed and names the rule responsible. If a result surprises you, your rules do not say what you think they say, and it is far cheaper to learn that now.
Step 5: Copy or download and deploy to the root
Copy the file or download robots.txt, then serve it at exactly https://yourhost/robots.txt with a 200 status and a text/plain content type. Nowhere else works. /blog/robots.txt is not read by anyone.
The Robots Exclusion Protocol, in detail
It is finally a standard
For twenty-five years robots.txt was a convention described in a 1994 mailing list post, and crawlers each interpreted the ambiguities their own way. In 2022 it was published as RFC 9309, which codifies the parsing rules, the matching semantics, and the precedence order that Google, Bing, and most serious crawlers had converged on. When this guide says "the rule is X", it means RFC 9309 says X — not that a blog post asserts it.
Groups, and why crawlers only obey one
A record consists of one or more consecutive User-agent: lines followed by the rule lines that apply to them. A crawler identifies itself with a product token — Googlebot, Bingbot, GPTBot — and looks for the group whose token most specifically matches. It obeys that one group and no other. The * group is the fallback, used only when nothing more specific matched.
The practical consequence bears repeating because it causes so much damage: groups do not inherit. A file that reads
User-agent: *
Disallow: /admin/
Disallow: /cart
User-agent: Googlebot
Disallow: /internal/
means Googlebot may crawl /admin/ and /cart freely. It found its own group, so the * group is invisible to it. If you want Googlebot blocked from all three, all three rules must appear inside the Googlebot group.
Longest-match precedence
When two or more rules in the applicable group match a URL, the winner is the one with the longest path pattern, measured in characters. Order in the file is irrelevant. If the longest matching Allow and the longest matching Disallow are the same length, Allow wins.
User-agent: *
Disallow: /admin
Allow: /admin/public
Under these rules, /admin/public/logo.png is allowed — the Allow pattern is 13 characters against the Disallow's 6 — while /admin/secret is disallowed, because only the Disallow pattern matches it. This is the mechanism behind every "block the directory, allow one file inside it" pattern, including WordPress's admin-ajax.php carve-out. It is also why writing rules top-down like a firewall config produces wrong intuitions: there is no first-match-wins, no fall-through, and no ordering.
Wildcards and the end-anchor
Two special characters are supported in path patterns.
* matches any sequence of characters, including none. Disallow: /*?sessionid= blocks any URL containing that query parameter anywhere.
$ anchors the end of the URL, and only means that when it is the final character of the pattern. Disallow: /*.pdf$ blocks /whitepaper.pdf but not /whitepaper.pdf?download=1, because that URL does not end at .pdf. Drop the $ and you block both — along with anything else whose path merely contains .pdf.
Without a $, matching is a prefix match, which trips people constantly. Disallow: /admin blocks /admin, /admin/, /admin/users, and also /administrator and /admin-tools, because all of them start with the string /admin. If you meant only the directory, write /admin/.
Crawl-delay is not universally honoured
Crawl-delay: 10 asks a crawler to wait ten seconds between requests. Bing, Yandex, and various smaller crawlers honour it. Googlebot ignores it completely — Google adjusts crawl rate based on server response times and the setting in Search Console, not on a directive in your file. Setting a large crawl delay on a large site is a slow-motion foot-gun: at 10 seconds per request, a 100,000-page site takes nearly twelve days to crawl once, and in practice much of it never gets crawled at all. If crawling is genuinely hurting your server, fix the server or cache the pages; throttling the crawler mostly just makes you invisible.
Blocking AI crawlers
The AI crawlers split into two categories that people routinely conflate. Training crawlers — GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended — gather content used to train models. Answer-engine crawlers, of which PerplexityBot is the clearest example, fetch pages so they can be cited in generated answers, which can send you referral traffic. Blocking the first category protects your content from being absorbed into a model; blocking the second removes you from a surface where you might otherwise be cited.
Google-Extended deserves a specific note because its naming misleads. It is not a crawler. It is a token that controls whether content already fetched by Googlebot may be used for Gemini and Vertex AI training. Disallowing it has no effect whatsoever on Google Search crawling, indexing, or ranking. You can refuse Google's AI training use and keep your Search presence entirely intact.
And the caveat that applies to all of them: compliance is voluntary. Blocking GPTBot stops OpenAI's declared crawler because OpenAI chooses to honour the file. It does nothing about a scraper that lies about its user-agent, and there is no robots.txt directive that can.
Directive reference
| Directive | Scope | Purpose | Honoured by |
|---|---|---|---|
User-agent: |
Opens a group | Names the crawler(s) the following rules apply to. * is the catch-all. |
Everyone |
Disallow: |
Inside a group | Asks the crawler not to fetch URLs matching the path. A bare Disallow: with no value means "allow everything". |
Everyone |
Allow: |
Inside a group | Carves an exception out of a broader Disallow. Wins ties by longest match. |
Google, Bing, most modern crawlers |
Crawl-delay: |
Inside a group | Minimum seconds between requests. | Bing, Yandex, others — not Googlebot |
Sitemap: |
Global | Absolute URL of a sitemap or sitemap index. Applies to all crawlers regardless of placement. | Google, Bing, most crawlers |
Host: |
Global | Preferred mirror/domain. A Yandex extension. | Yandex only |
Noindex: |
— | Does not work. Google dropped support in 2019. Use a noindex meta tag or X-Robots-Tag header. |
Nobody |
# |
Anywhere | Comment. Everything after it on the line is ignored. | Everyone |
Common use cases
Keeping junk out of the index without blocking anything important
The bread-and-butter job: block faceted-search URLs, session-ID query strings, internal search results, and cart or checkout pages so crawlers spend their budget on pages that can actually rank. The trap is over-blocking. A rule like Disallow: /*? blocks every URL with a query string, which on many sites includes paginated category pages you very much want crawled. Test the patterns before shipping them.
Taking a staging site dark
User-agent: * plus Disallow: / is the right call for a staging or preview environment, and the Block everything preset produces it. But treat this as hygiene, not security: staging environments should also be behind HTTP auth or an IP allowlist, because robots.txt neither hides the host nor stops anyone from browsing it. And the file must never, ever be promoted to production — put it in the deploy pipeline's diff review, because this exact mistake is the single most common way sites disappear from Google.
Fixing a site that has dropped out of search
When organic traffic falls off a cliff, /robots.txt is the first thing to check, before the algorithm updates and the backlink audits. Paste the live file into the generator and read the validator output. A stray Disallow: /, a rule that blocks the CSS and JavaScript Googlebot needs to render the page, or a Googlebot group that accidentally overrides a carefully written * group will show up immediately.
Deciding what AI crawlers may do with your content
Publishers, documentation sites, and anyone whose content is the product now have a real decision to make about training crawlers. The Block AI crawlers preset gives you a defensible default — search engines welcome, training crawlers refused — and the crawler table explains each one so you can make exceptions deliberately, such as keeping PerplexityBot allowed for the referral traffic while refusing the pure training bots.
Auditing an inherited site
You take over a site and nobody knows why /resources/ is not indexed. Import the live robots.txt, run the tester against the paths in question, and the tool names the exact rule doing it. This takes a minute and replaces an afternoon of guessing.
Why generate robots.txt in the browser
The Robots.txt Generator runs entirely client-side. Your paths, hostnames, and rules never leave the machine, which matters more than it might seem — the directory structure of an unreleased product, the URL of a staging host, and the internal routes you are trying to keep quiet are all things worth not pasting into someone else's server logs. Once the page is loaded it works offline, and the output is a plain text file you own.
Robots.txt sits alongside a few neighbouring jobs. The Meta Tag Generator produces the noindex and canonical tags that do the work robots.txt cannot. The Open Graph Preview shows how your links will render once those crawlers you did allow come and fetch them. And the Slug Generator builds the clean paths your rules will end up matching against.
FAQ
Where do I put the robots.txt file?
It must be served at exactly /robots.txt on the host it applies to — for example https://example.com/robots.txt. Crawlers look nowhere else. A file at /blog/robots.txt is ignored entirely, and the file on example.com does not govern shop.example.com; each host needs its own. Serve it with a 200 status and a text/plain content type.
Does Disallow remove a page from Google?
No, and this is the most consequential misunderstanding about the format. Disallow stops Google from fetching a page; it does not remove the URL from the index. If other sites link to a blocked URL, Google can still list it, usually with no description because it never read the page. To keep a page out of search results, allow crawling and serve a noindex robots meta tag or an X-Robots-Tag header. If you block the URL, the crawler can never see the noindex you added.
Is robots.txt a way to hide private pages?
No. It is a public file that anyone can read, and honouring it is voluntary — malicious scrapers ignore it completely. Listing /internal-admin/ in your Disallow rules tells every attacker exactly where to look. Anything that must stay private needs authentication, an IP allowlist, or to not be on the public internet at all.
How do Allow and Disallow interact when both match a URL?
The most specific rule wins, where specificity means the length of the path pattern. Given Disallow: /admin and Allow: /admin/public, the URL /admin/public is allowed because the Allow pattern is longer, while /admin/secret is blocked. If two patterns are exactly the same length, Allow wins. The order rules appear in the file does not matter, which surprises people who expect firewall-style first-match-wins behaviour.
What do the * and $ wildcards do?
An asterisk matches any run of characters, so Disallow: /*?sessionid= blocks any URL containing that parameter. A dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?download=1. Without a $, matching is a prefix match: Disallow: /admin blocks /admin, /admin/users, and also /administrator, because all of them begin with that string.
How do I block AI crawlers like GPTBot and ClaudeBot?
Give each one its own group with Disallow: /. The Block AI crawlers preset does this for GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, and anthropic-ai in a single click while leaving Googlebot and Bingbot free to crawl. Note that Google-Extended only controls training use for Gemini and Vertex AI and has no effect on Google Search. Compliance is voluntary, so this stops cooperative crawlers, not determined scrapers.
Does Crawl-delay actually work?
It depends on the crawler. Googlebot ignores it completely — crawl rate is adjusted from Search Console and from your server's response times. Bing, Yandex, and several smaller crawlers do honour it, treating the value as the minimum number of seconds between requests. Setting a large delay on a big site can mean most pages never get crawled at all, so keep values small and fix server capacity instead if crawling is genuinely a problem.
What happens if my robots.txt has a syntax error?
Crawlers silently discard lines they cannot parse and carry on with the rest, so a broken rule fails quietly rather than loudly. An unknown directive, a path missing its leading slash, or a rule placed above the first User-agent: line simply does nothing, and you will not find out until your traffic moves. That is exactly what the validator is for: it reports each of these with a line number before you deploy.
