Robots.txt Generator
Build a robots.txt file by choosing which paths crawlers may access and pointing them to your sitemap. Copy or download the result.
How to use robots.txt generator
- Set the user-agent
Use * for all crawlers or name a specific bot.
- List allow/disallow paths
Add the paths you want to permit or block, one per line.
- Generate the file
Click Generate, then copy or download robots.txt to your site root.
Features
Fine-tune exactly which sections of your site get crawled.
Include your sitemap URL so search engines find all your pages.
One checkbox produces a rule that blocks all crawlers for staging sites.
Frequently asked questions
Where does robots.txt go?
It must live at the root of your domain, for example https://example.com/robots.txt.
Does robots.txt guarantee pages stay private?
No. It requests that compliant crawlers skip paths, but it doesn't hide content. Use authentication for anything truly private.
Should I block a whole site?
Only for staging or private environments. Blocking all crawlers on a live site will remove it from search results.