Robots.txt Generator

A Robots.txt Generator builds the plain-text file that sits at the root of your domain and tells crawlers which parts of your site they may fetch. The format looks simple — a `User-agent:` line followed by `Allow:` and `Disallow:` rules — but the details bite: rules only apply inside a group, paths must begin with a slash, `*` and `$` behave in specific ways, and a single stray `Disallow: /` under `User-agent: *` can pull an entire site out of Google. This tool builds the file from a structured form so the syntax is correct by construction.

It does three jobs beyond generation. It lints your file and flags the real footguns: rules floating outside any group, relative paths, a bare `Disallow:` that silently means "allow everything", sitemap URLs that are not absolute, unknown directives, and blanket blocks. It parses an existing robots.txt back into editable groups, so you can paste what is live today and fix it. And it tests a URL against your rules with the same longest-match precedence real crawlers use — the most specific matching pattern wins, and Allow beats Disallow on ties — so you can prove a page is reachable before you deploy.

One thing worth stating plainly: robots.txt controls crawling, not indexing, and it is not a security boundary. A blocked URL can still appear in search results if other pages link to it, because Google can index a URL it has never fetched. To keep a page out of the index you need a `noindex` meta tag or an X-Robots-Tag header — which the crawler can only see if it is allowed to fetch the page. And the file is public: anyone can read yours at /robots.txt, so listing a secret directory there advertises it. Use authentication for anything private.