Robots.txt Generator
Create a robots.txt file that tells search engines which parts of your site to crawl. Tick the paths to block, add your sitemap URL, and download a ready-to-upload file.
Use * to apply the rules to all search engines, or name one (for example Googlebot).
Your result will appear here
Your robots.txt
Upload this file to the root of your site (for example https://example.com/robots.txt). Note: robots.txt asks crawlers not to visit pages — it does not hide them from people who know the URL.
How to use
- Choose the user-agent the rules apply to — * covers all search engines.
- Tick the paths you want to block, add any custom paths, and paste your sitemap URL if you have one.
- Click Generate robots.txt, then Download robots.txt and upload the file to your site's root.
Frequently asked questions
What is robots.txt?
It is a plain-text file at the root of a website (for example https://example.com/robots.txt) that tells search-engine crawlers which pages or folders they may or may not visit.
Does robots.txt hide pages from Google?
Not reliably. It asks well-behaved crawlers not to visit a page, but the page can still be indexed if other sites link to it, and anyone who knows the URL can still open it. For truly private content, use a password or a noindex tag instead.
What is the difference between Disallow and Allow?
Disallow tells crawlers not to visit a path; Allow makes an exception inside a disallowed folder. This generator uses Disallow rules only, which covers the most common cases.
Where do I upload robots.txt?
Upload it to the root of your domain, so it is reachable at https://yourdomain.com/robots.txt. A robots.txt in a subfolder has no effect.