Robots.txt Generator
Build a robots.txt file with a separate rule group per bot — allow/disallow paths, crawl-delay, and one or more sitemap references. Includes one-click presets for Googlebot, Bingbot and common AI crawlers. Copy or download the result.
How to use robots.txt generator
- Add one or more user-agent groups
Each group targets a bot (* for all, or a specific crawler) with its own allow/disallow paths and optional crawl-delay.
- Use a preset or add your own
Quick-add Googlebot, Bingbot or a block-all-AI-crawlers group, or build a custom one.
- Generate the file
Click Generate, then copy or download robots.txt to your site root.
Features
Give Googlebot, Bingbot, AI crawlers or any bot their own separate rules, not just one blanket group.
Set an optional crawl-delay per group for bots that respect it (Googlebot ignores it — use Search Console instead).
List one or more sitemap URLs so search engines find all your pages.
One click adds a rule group blocking common AI training crawlers (GPTBot, CCBot, Google-Extended and others).
Supported formats and limitations
Text and structured input supported by this tool.
Use cases include personal workflows, content preparation, and quick checks. Review output before relying on it for legal, financial, accessibility, or production decisions.
Frequently asked questions
Where does robots.txt go?
It must live at the root of your domain, for example https://example.com/robots.txt.
Does robots.txt guarantee pages stay private?
No. It requests that compliant crawlers skip paths, but it doesn't hide content. Use authentication for anything truly private.
Should I block a whole site?
Only for staging or private environments. Blocking all crawlers on a live site will remove it from search results.
Does crawl-delay actually slow Googlebot down?
No — Google ignores the Crawl-delay directive entirely. It's respected by some other bots (like Bingbot), but to control Googlebot's crawl rate you need Google Search Console instead.
Can I block AI crawlers like ChatGPT's without blocking search engines?
Yes — that's exactly what the 'Block AI crawlers' preset adds: a separate rule group naming known AI-training bots (GPTBot, CCBot, Google-Extended, and others) with Disallow: /, left alongside your normal Googlebot/Bingbot/* groups untouched.