🤖

Free Robots.txt Generator — Build Crawl Rules & Download

Set allow/disallow rules per user-agent, add sitemap URLs, and download your robots.txt file instantly. Includes presets for common configurations.

Quick Presets

Apply a preset to auto-fill rules below

Edit Rules

Sitemap URL

Sitemap URL Optional — one per line

Generated robots.txt

Save as robots.txt and upload to your site's root directory.

💡 Robots.txt Tips

robots.txt must be placed at your site root (e.g., https://example.com/robots.txt).
Disallow: / blocks the entire site for that user-agent. Double-check before using this!
Allow: can create exceptions within a disallowed directory — more specific rules take precedence.
robots.txt is not a security measure. Pages listed in robots.txt can still be found if linked from elsewhere. Use authentication for truly private pages.
Test your robots.txt with Google's robots.txt Tester in Search Console.

Frequently Asked Questions

Does blocking a URL in robots.txt remove it from Google search results? ▾

Not completely. Disallow prevents crawling but not indexing. If the URL has external links pointing to it, Google may still index the URL (showing it without a snippet). To fully remove a page from search results, use <meta name="robots" content="noindex"> in combination.

What does "User-agent: *" mean? ▾

The asterisk (*) is a wildcard that applies rules to all web crawlers. To set different rules for specific crawlers (like Googlebot or Bingbot), add separate User-agent blocks with the crawler's name.

Should I block AI crawlers in my robots.txt? ▾

This is a personal/business decision. If you don't want your content used for AI training, you can block GPTBot, CCBot, anthropic-ai, and Google-Extended. Note that not all AI crawlers respect robots.txt, and this won't affect your Google search rankings.

How quickly does Google pick up robots.txt changes? ▾

Google typically re-fetches robots.txt multiple times per day. Use the robots.txt tester in Google Search Console to immediately validate your changes without waiting for Googlebot to recrawl.