robots.txt Generator
Write correct crawl directives without guessing at syntax. Includes a WordPress preset, AI crawler blocking and a warning when you are about to hide your whole site.
Paths are relative to the domain root and are case sensitive. * matches any characters, $ anchors the end of a URL.
Use these to carve exceptions out of a broader Disallow rule.
Google ignores this. Bing and Yandex respect it.
robots.txt
Upload to the root of your domain: https://example.com/robots.txt. It will not work in a subfolder.
Common mistakes
- Disallow is not noindex.A blocked URL can still appear in results without a snippet. To remove a page, allow crawling and use a
noindexmeta tag. - Do not block CSS or JS.Google renders pages; blocking assets breaks how it sees your layout.
- Case matters.
/Admin/and/admin/are different rules.
How crawlers read the file
Each crawler looks for the group whose User-agent best matches its own name, then follows only that group. It does not merge groups. If you write a specific Googlebot block, Googlebot ignores the * block entirely — so repeat any shared rules inside the specific group.
When two rules conflict, the most specific match wins, and Allow beats Disallow at equal length. An empty Disallow: line means "allow everything", which is why a blank value is safer than a missing file.