Free Robots.txt Generator
Generate a robots.txt file for your website. Set user-agent rules, allow/disallow paths, add sitemap URL, and configure crawl delay. Download or copy instantly. No signup required.
100% free — no account needed
Presets
Preview
User-agent: *
Free Robots.txt Generator Features
Tell search engines what to index
User-Agent Rules
Set rules for all crawlers (*), Googlebot, Bingbot, or custom user agents.
Copy & Download
Copy to clipboard or download as a robots.txt file.
3 Presets
Allow all, block all, or block sensitive folders (/admin, /api, /private).
Allow & Disallow
Add multiple allow and disallow path rules for precise control.
FAQ
What is robots.txt?
A text file at your website root (example.com/robots.txt) that tells search engine crawlers which pages to crawl and which to skip.
Does robots.txt block indexing?
No. It only controls crawling. Pages can still be indexed if linked from elsewhere. Use meta noindex to prevent indexing.
Should I include a sitemap?
Yes. Adding your sitemap URL in robots.txt helps search engines discover all your pages efficiently.
Optimize Your SEO
Meta tags, robots.txt, and content scheduling — all in one place.