Free tools · No signup

AI crawler rules tester

Find which crawlers your robots.txt allows, with the exact rule behind each result.

← All free tools

Your input is processed in this browser. Text and files are not uploaded or saved by this tool. Use up to 200,000 characters per text area.

Only the URL path is tested against your pasted rules. No website is fetched.
An empty file means no declared restrictions. Paste up to 200,000 characters.

Your result

Enter your details or load an example, then run the tool.

What this tool can tell you

Checks supplied rules only. Firewall access, actual indexing and AI citations cannot be established by this test.

Need evidence across your website? Explore CiteLadder’s site-wide workflow.