How to Write a robots.txt File
By Tools Star Hub Editorial Team · Published
robots.txt is text a site can publish for crawlers. This page builds that text from user-agent groups and an optional sitemap URL. Copying the result does not put the file on a website, and the page does not request your site.
Each group starts with a User-agent line, then one Allow line per path and one Disallow line per path. Blank path lines are skipped. A sitemap line is added only when the address is an absolute http or https URL.
How to write a robots.txt file
- Enter a user-agent. An asterisk means every crawler. Each group needs a user-agent. An empty group is rejected.
- Put paths on their own lines. Allow paths and disallow paths are separate boxes. A blank line is skipped.
- Add another group when a second crawler needs different rules. Each group has its own user-agent and its own allow and disallow lines.
- Add a sitemap only if you want one. example.com/sitemap.xml is rejected because it has no protocol. Press Generate and copy the text. You still have to place that file at the site root yourself.
Example
One disallow rule
One group with user-agent * and disallow /admin, and no sitemap, is User-agent: * followed by Disallow: /admin.
Several rules and a sitemap
Allow paths /public and /assets, disallow paths /admin and /private, and sitemap https://example.com/sitemap.xml write those four rules and end with Sitemap: https://example.com/sitemap.xml. A second group for Bingbot with allow / is included, and a blank sitemap box adds no Sitemap line. An allow list of a blank row and /ok becomes only Allow: /ok.
FAQ
- Does the page publish the file?
- No. It writes text you can copy. A crawler sees the rules only after the file is on the site.
- Can two groups use different agents?
- Yes. Each group has its own user-agent and its own allow and disallow lines.
- What does a blank path do?
- It is skipped. It does not become an empty Allow or Disallow line.
- Which sitemap addresses work?
- An absolute http or https URL with a hostname. A bare domain is rejected.
To build robots.txt text you can copy, use the Robots.txt Generator.