How do you write a robots.txt file?

Start with a User-agent line, then add Allow and Disallow lines. Add a Sitemap line when you have an absolute sitemap URL.

Can I use more than one user-agent?

Yes. Each group has its own user-agent and rules.

What happens to a blank path?

A blank line is skipped, so it does not create an empty Allow or Disallow rule.

Does this test my live site?

No. It only generates the text. It does not publish the file or request your site.

What sitemap URL is accepted?

An absolute http or https URL. A path without a protocol is rejected.

Are these numbers sent to a server?

No. The user-agent lines and paths are assembled in this tab. They are not uploaded, and the page does not request your site.

SEO & Utilities

Robots.txt Generator

Build robots.txt text from user-agent, allow, and disallow rules, plus an optional sitemap URL.

What is Robots.txt Generator?

A robots.txt generator writes User-agent, Allow, and Disallow lines from the rules you type. It can add a sitemap URL. It does not publish or test a live site.

This writes robots.txt text from the rules you type. It does not test or publish a live site.

Group 1 user-agent

Allow paths, one per line

Disallow paths, one per line

Sitemap URL, optional

Loading tool…

This tool runs in your browser. Inputs, files and generated values stay on this device. Favorites and recently used tools, if you use them, store only tool names in local storage — never passwords, documents or QR contents. See the privacy policy.

Robots.txt Generator — Allow, Disallow, and Sitemap Rules — Tools Star Hub

What this tool does

Write robots.txt text from one or more user-agent groups and an optional sitemap URL.

How to use

Enter a user-agent.
Add allow and disallow paths, one per line.
Add a sitemap URL if you want one.
Press Generate.

Examples

A private folder
User-agent * with Disallow: /admin tells crawlers not to fetch paths under /admin. The file is text only.

Main features

Several user-agent groups.
Several allow and disallow lines.
An optional absolute sitemap URL.

How it works

Each group starts with User-agent, then one Allow line for each path and one Disallow line for each path. Blank path lines are skipped. A sitemap is added only when it is an absolute http or https URL. The page does not upload the file or test a live site.

Tips

Use * for every crawler.
Put each path on its own line.

Limitations

The result is text you can copy. It does not publish rules or check what a live site allows.

See the disclaimer for what these tools do not cover.

FAQ

How do you write a robots.txt file?: Start with a User-agent line, then add Allow and Disallow lines. Add a Sitemap line when you have an absolute sitemap URL.
Can I use more than one user-agent?: Yes. Each group has its own user-agent and rules.
What happens to a blank path?: A blank line is skipped, so it does not create an empty Allow or Disallow rule.
Does this test my live site?: No. It only generates the text. It does not publish the file or request your site.
What sitemap URL is accepted?: An absolute http or https URL. A path without a protocol is rejected.
Are these numbers sent to a server?: No. The user-agent lines and paths are assembled in this tab. They are not uploaded, and the page does not request your site.

URL Parser

SEO & Utilities

Split an absolute URL into protocol, host, port, path, fragment, and query parameters.

UTM Builder

SEO & Utilities

Add utm_source, utm_medium, and utm_campaign to a link. Term and content are optional. Blank fields are omitted, and this page does not record a visit.

Slug Generator

SEO & Utilities

Turn a title into a lowercase, hyphenated permalink. Latin accents are removed, while letters such as 你好 stay. It does not check that the address is unused.