Robots.txt Generator
Generate a clean robots.txt for your site, with sitemap reference, AI crawler controls, and platform-specific disallow rules.
Your robots.txt
Frequently asked questions
Where does robots.txt go? +
At the root of your domain, example.com/robots.txt. It MUST be at the root, not in a subfolder.
Can I block AI crawlers? +
Yes, major AI crawlers honour robots.txt: GPTBot, ClaudeBot, Google-Extended, CCBot, anthropic-ai. Whether to block them is a strategic call (visibility vs training).
What's the difference between robots.txt and noindex? +
robots.txt blocks crawling. noindex blocks indexing. A page blocked by robots.txt that's already indexed will stay in the index, to remove it, use noindex first, then block.