Robots.txt Generator
Build a robots.txt with user-agent groups, Allow and Disallow paths, crawl-delay and Sitemap lines, written to RFC 9309 syntax and ready to download.
The interactive tool loads here when JavaScript runs.
About this toolWhat is the Robots.txt Generator?
The Robots.txt Generator builds the plain-text file that tells crawlers which paths they may fetch. Add one group per user agent (the default is *, meaning every crawler), list Disallow and Allow paths, optionally set a crawl delay, add Sitemap URLs and click Generate Robots.txt. Copy the result or download it as robots.txt.
The output follows the syntax in RFC 9309. Paths that do not start with / or * are given a leading slash so they match what you meant, duplicate lines are removed, and a group with no rules gets an explicit empty "Disallow:" so every parser reads it as "allow everything". Sitemap lines go at the end, outside the groups, because they apply to the whole file.
Everything is generated in your browser. The tool does not test URLs against the rules or fetch your current robots.txt; use the robots.txt report in Google Search Console for that.
- Multiple user-agent groups with Allow and Disallow paths
- RFC 9309 path normalisation and duplicate removal
- Per-group Crawl-delay and multiple Sitemap lines
- Common disallow paths in one click
- Copy or download as robots.txt; runs locally
Updated
How to use the Robots.txt Generator
- In User Agent Rules, keep * for all crawlers or type a specific name such as Googlebot or GPTBot. Use Add User Agent Rule for more groups.
- Click the + next to Disallow Paths or Allow Paths to add a path, or use Add common disallow paths for typical admin and temp folders.
- Optionally set Crawl Delay (seconds) for crawlers that honour it, and add one or more Sitemap URLs under Additional Settings.
- Click Generate Robots.txt, then Copy or Download, and upload the file to the root of the host, for example https://example.com/robots.txt.
Worked examples
Block a folder but keep one subfolder crawlable
Input
User agent *, Disallow /admin/, Allow /admin/public/, Sitemap https://example.com/sitemap.xml
Output
User-agent: * Disallow: /admin/ Allow: /admin/public/ Sitemap: https://example.com/sitemap.xml
/admin/public/logo.png matches both rules; the Allow path is longer (14 characters against 7), so under RFC 9309 it is allowed. /admin/users is blocked.
Path typed without a leading slash
Input
Disallow path: private/
Output
Disallow: /private/
RFC 9309 paths start with /. The generator adds it so the rule is not ignored.
Background and insight
The Robots Exclusion Protocol was proposed by Martijn Koster in 1994 and stayed an informal convention for 28 years until the IETF published it as RFC 9309 in September 2022. The RFC settles the details crawlers used to disagree on. Rules are prefix matches on the URL path, * matches any sequence of characters and $ anchors the end. When several rules match, the one with the longest path wins, and if an Allow and a Disallow are equally long, Allow wins.
That longest-match rule is why order does not matter. With "Disallow: /admin/" and "Allow: /admin/public/", the URL /admin/public/logo.png is allowed because the Allow pattern is longer, wherever the lines appear. A crawler obeys only the most specific group that names it, so a "User-agent: Googlebot" group replaces the * group for Googlebot rather than adding to it.
Robots.txt controls crawling, not indexing or access. A disallowed URL can still be indexed without its content if other sites link to it, and anyone can read the file, so listing /secret-admin-panel/ advertises it. Google never supported noindex inside robots.txt and stopped honouring the unofficial rule in 2019. Crawl-delay is not in the RFC: Bing and Yandex honour it, Googlebot ignores it. Host was a Yandex-only directive that Yandex itself retired in 2018.
Common use cases
Keeping crawlers out of faceted URLs
Disallow /search or ?sort= style URLs that create endless near-duplicate pages.
Opting out of AI training crawlers
Add groups for GPTBot, Google-Extended or CCBot with Disallow: / while leaving search crawlers alone.
Staging environments
Block everything on a staging host, and protect it with authentication as well.
Pointing crawlers at sitemaps
List every sitemap or sitemap index with an absolute URL.
Tips and limitations
- robots.txt is per host and protocol. https://www.example.com/robots.txt does not cover https://shop.example.com.
- Never rely on robots.txt for privacy. Use authentication, and use noindex (on a crawlable page) to keep pages out of results.
- Do not block CSS and JavaScript that pages need to render; Google renders pages and judges them as users see them.
- Google caches robots.txt for up to about 24 hours and reads at most 500 KiB of it, so keep the file short.
- If robots.txt returns a 5xx error, Google treats the whole site as disallowed until it can read the file again.
Sources and further reading
- RFC 9309: Robots Exclusion Protocol, IETF. The 2022 standard, including the longest-match rule
- A Standard for Robot Exclusion, The Web Robots Pages. Martijn Koster's original 1994 document
- How Google interprets the robots.txt specification, Google Search Central. Google's handling of groups, wildcards and unsupported lines
- A note on unsupported rules in robots.txt, Google Search Central Blog. The 2019 end of noindex in robots.txt
Frequently asked questions
Which rule wins when Allow and Disallow both match a URL?
The rule with the longest matching path wins, following RFC 9309. If an Allow and a Disallow match with paths of equal length, Allow wins. The order of lines in the file does not matter, so "Allow: /admin/public/" beats "Disallow: /admin/" for any URL under /admin/public/.
Does Disallow in robots.txt remove a page from Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results, usually without a snippet, if other pages link to it. To remove a page, let Google crawl it and return a noindex robots meta tag or header, or remove the page and return 404 or 410.
Does Googlebot respect Crawl-delay?
No. Crawl-delay is not part of RFC 9309 and Googlebot ignores it; Google adjusts crawl rate automatically based on how your server responds. Bing and Yandex do honour it. If Googlebot is overloading a server, return 503 or 429 temporarily rather than adding a delay line.
Is a robots.txt Disallow a prefix match?
Yes. "Disallow: /blog" blocks /blog, /blog/, /blog/post and also /blogging-tips, because each starts with /blog. Add a trailing slash to block only the folder, or use $ to anchor the end, as in "Disallow: /*.pdf
quot; for URLs ending in .pdf.How do I block AI crawlers with robots.txt?
Add a group per crawler with "Disallow: /", for example User-agent: GPTBot, User-agent: CCBot or User-agent: Google-Extended. Google-Extended controls use for Gemini training without affecting Google Search. These bots choose to honour robots.txt; it is a request, not an enforcement mechanism.
Where must the robots.txt file be placed?
At the root of each host and protocol, as /robots.txt, for example https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own. It must be plain text, served with a 200 status, and UTF-8 encoded.
Should the Sitemap line be inside a User-agent group?
No. Sitemap lines are independent of groups and apply to every crawler, so the generator writes them at the end of the file. Use an absolute URL, and list each sitemap or sitemap index on its own line.
Is there an in-depth guide to the Robots.txt Generator?
Yes. “Robots.txt and XML Sitemaps: How Crawlers Actually Read Them” covers the background, how it works, common pitfalls and worked examples. Read the full guide.
Related Tools You May Like
Free, in-browser tools that pair well with this one.