Turn titles into clean URL slugs with Arabic transliteration, diacritic removal, stop words and a length limit
robots.txt Generator & Tester
Generate a robots.txt from templates (WordPress, block AI crawlers) and test any URL against its rules using the longest-match algorithm of RFC 9309, with warnings for blocking the whole site or CSS/JS files.
Workspace
1) Build the file
2) Editor
3) Test a URL
How the result is calculated
The file is parsed line by line (comments after # are removed) and consecutive User-agent lines form one group. For a bot the group whose name equals its name (case-insensitive) is chosen, otherwise the * group. Allow/Disallow rules are then matched per RFC 9309: prefix matching on the path plus query, * matches any sequence, a trailing $ anchors the match, and encodings are normalized (%xx uppercase, UTF-8 for non-ASCII). The longest matching rule wins and Allow wins a tie. An empty Disallow: has no effect, /robots.txt is always allowed, and no matching rule means allowed.
Limits:
The tool implements RFC 9309, not the behaviour of every crawler: Google also falls back from a specialised bot (such as Googlebot-Image) to Googlebot, which is not applied here, so test with the exact bot name. The AI crawler names in the template change over time; check each vendor’s documentation. robots.txt does not prevent indexing (a blocked URL can still be indexed if linked from elsewhere); use noindex for that. The file is not fetched from your site: paste it into the editor.
FAQ
Does robots.txt keep a page out of Google?
No, it blocks crawling, not indexing. A blocked URL can still appear without a description if other pages link to it. Use noindex to actually exclude a page, and do not block it in robots.txt so Google can see the tag.
Which rule wins when they conflict?
The longest match; if tied, Allow. For example Disallow: /p and Allow: /page allow /page.
Does Google ignore Crawl-delay?
Yes, Google ignores it, while some other crawlers use it. To manage Google’s crawl rate use Search Console tools.
