Robots.txt Generator

Generate robots.txt with a visual editor. Presets for standard sites, WordPress, e-commerce, and AI crawler blocking. Add sitemaps and crawl delays.

What is Robots.txt Generator?

A robots.txt file tells crawlers which pages to access. This generator helps create it correctly with rules for different bots, crawl delays, and sitemaps.

How to Use Robots.txt Generator

Start with a preset or build from scratch. Add rules with user agents and paths. Add sitemap URLs. Copy or download the result.

FAQ

Where do I place robots.txt?

In your website's root directory (e.g., https://example.com/robots.txt).

Can I block AI crawlers?

Yes, use the Block AI Crawlers preset for GPTBot, ClaudeBot, Google-Extended, and others.

Does robots.txt prevent indexing?

No, it's a directive, not enforcement. Use noindex meta tags for stronger protection.