robots.txt Generator & Tester

Generate a robots.txt from templates (WordPress, block AI crawlers) and test any URL against its rules using the longest-match algorithm of RFC 9309, with warnings for blocking the whole site or CSS/JS files.

Workspace

1) Build the file

2) Editor

3) Test a URL

How the result is calculated

The file is parsed line by line (comments after # are removed) and consecutive User-agent lines form one group. For a bot the group whose name equals its name (case-insensitive) is chosen, otherwise the * group. Allow/Disallow rules are then matched per RFC 9309: prefix matching on the path plus query, * matches any sequence, a trailing $ anchors the match, and encodings are normalized (%xx uppercase, UTF-8 for non-ASCII). The longest matching rule wins and Allow wins a tie. An empty Disallow: has no effect, /robots.txt is always allowed, and no matching rule means allowed.

Limits: The tool implements RFC 9309, not the behaviour of every crawler: Google also falls back from a specialised bot (such as Googlebot-Image) to Googlebot, which is not applied here, so test with the exact bot name. The AI crawler names in the template change over time; check each vendor’s documentation. robots.txt does not prevent indexing (a blocked URL can still be indexed if linked from elsewhere); use noindex for that. The file is not fetched from your site: paste it into the editor.

FAQ

Does robots.txt keep a page out of Google?

No, it blocks crawling, not indexing. A blocked URL can still appear without a description if other pages link to it. Use noindex to actually exclude a page, and do not block it in robots.txt so Google can see the tag.

Which rule wins when they conflict?

The longest match; if tied, Allow. For example Disallow: /p and Allow: /page allow /page.

Does Google ignore Crawl-delay?

Yes, Google ignores it, while some other crawlers use it. To manage Google’s crawl rate use Search Console tools.

Newest tools

Recently added free tools — they run in your browser without uploading your data

All tools →

AES and ChaCha20 keys with correct IV/nonces via Web Crypto, Hex, Base64 and JWK formats, verification and bulk export

Local

Paste discussions (Reddit, Quora, YouTube) to extract pain points, repeated questions, phrases, topic clusters and title ideas

Local

A bookmarklet that analyses any page as your browser sees it after JavaScript: title, description, H1, links, images, Open Graph, JSON-LD and hreflang

Local

Legitimate link-building tactics, outreach message templates, a linkable-asset idea generator and a local opportunity tracker with CSV and JSON export

Local