Robots.txt Generator
Generate a valid robots.txt with allow/disallow rules and sitemap references.
Examples
See what this tool can do.
Block a folder
Disallow /admin/ while allowing the rest of the site.
Point to your sitemap
Add your sitemap URL so crawlers find it quickly.
Tips & tricks
Get the most out of it.
- 1Always include your sitemap URL to help discovery.
- 2Test rules before deploying — a stray disallow can hide your whole site.
- 3robots.txt is public; don't list secret paths you want hidden.
About Robots.txt Generator
What this tool does
Robots.txt Generator builds a valid robots.txt file with allow/disallow rules, user-agent groups and a sitemap reference.
How to use it
- Add user-agent groups with their allow/disallow paths.
- Optionally add your sitemap URL and host.
- Copy or download the generated robots.txt.
How it works
Your rules are assembled into the standard robots.txt format, grouped by user-agent, with correct directive ordering and an optional Sitemap line.
Important considerations
- robots.txt guides crawlers but doesn't secure pages — use noindex or authentication to keep pages out of search or private.
Privacy
This tool runs entirely in your browser. The text, HTML or settings you enter are processed locally and never uploaded to a server.
Related SEO tools
Frequently asked questions
Not reliably. It controls crawling, not indexing. To keep a page out of results, use a noindex tag; to keep it private, require authentication.