A text file at your website root (example.com/robots.txt) that tells search engine crawlers which pages to crawl and which to skip.

Does robots.txt block indexing?

No. It only controls crawling. Pages can still be indexed if linked from elsewhere. Use meta noindex to prevent indexing.

Should I include a sitemap?

Yes. Adding your sitemap URL in robots.txt helps search engines discover all your pages efficiently.

SEO Tool

Robots.txt Generator

Generate a robots.txt file for your website. Set user-agent rules, allow/disallow paths, add sitemap URL, and configure crawl delay. Download or copy instantly.

100% free — no account needed

Presets

User-Agent

Rules

Sitemap URL

Crawl Delay (seconds)

Preview

User-agent: *

Control Search Crawlers

Tell search engines what to index

User-Agent Rules

Set rules for all crawlers (*), Googlebot, Bingbot, or custom user agents.

Allow & Disallow

Add multiple allow and disallow path rules for precise control.

3 Presets

Allow all, block all, or block sensitive folders (/admin, /api, /private).

Copy & Download

Copy to clipboard or download as a robots.txt file.

FAQ

SEO Tools

Optimize Your SEO

Meta tags, robots.txt, and content scheduling — all in one place.