Robots.txt generator
Build a robots.txt file that tells search engine crawlers which parts of your site they may visit.
How to use the robots.txt generator
- Keep the default rule for all crawlers or add specific ones.
- List paths to block (like /admin) and your sitemap URL.
- Download robots.txt and upload it to your site's root.
When it's useful
- Blocking admin or checkout pages from crawling
- Telling crawlers where your sitemap is
- Creating a first robots.txt for a new site
Limits to know
- robots.txt asks well-behaved crawlers not to visit; it doesn't hide pages or protect private data.
- To keep a page out of Google, use a noindex tag instead — blocked pages can still be indexed from links.
Your privacy
Everything is worked out in your browser. What you type isn't uploaded, saved or shared, and there's no sign-up.
Questions
Where do I put robots.txt?
At the root of your domain, e.g. https://example.com/robots.txt.
Does Disallow remove a page from Google?
No. It stops crawling. Use noindex to remove a page from results.