AI crawler rules tester

Find which crawlers your robots.txt allows, with the exact rule behind each result.

Your input is processed in this browser. Text and files are not uploaded or saved by this tool. Use up to 200,000 characters per text area.

Your input

Only the URL path is tested against your pasted rules. No website is fetched.

An empty file means no declared restrictions. Paste up to 200,000 characters.

Enter your details or load an example, then run the tool.

Your result

Enter your details or load an example, then run the tool.

What this tool can tell you

Checks supplied rules only. This test cannot establish firewall access, actual indexing or AI citations.

Need evidence across your website? Site Health inspects captured page evidence before you choose a fix.

Explore Site Health