Generators

robots.txt Generator

Write correct crawl directives without guessing at syntax. Includes a WordPress preset, AI crawler blocking and a warning when you are about to hide your whole site.

Paths are relative to the domain root and are case sensitive. * matches any characters, $ anchors the end of a URL.

Use these to carve exceptions out of a broader Disallow rule.

Google ignores this. Bing and Yandex respect it.

robots.txt


        

Upload to the root of your domain: https://example.com/robots.txt. It will not work in a subfolder.

Common mistakes

  • Disallow is not noindex.A blocked URL can still appear in results without a snippet. To remove a page, allow crawling and use a noindex meta tag.
  • Do not block CSS or JS.Google renders pages; blocking assets breaks how it sees your layout.
  • Case matters./Admin/ and /admin/ are different rules.

How crawlers read the file

Each crawler looks for the group whose User-agent best matches its own name, then follows only that group. It does not merge groups. If you write a specific Googlebot block, Googlebot ignores the * block entirely — so repeat any shared rules inside the specific group.

When two rules conflict, the most specific match wins, and Allow beats Disallow at equal length. An empty Disallow: line means "allow everything", which is why a blank value is safer than a missing file.

Related tools