Free tools · No signup

Robots.txt generator

Choose crawler permissions, add your sitemap and download a tested starting point.

← All free tools

Your input is processed in this browser. Text and files are not uploaded or saved by this tool. Use up to 200,000 characters per text area.

Block selected crawlers

Unchecked crawlers follow the general path rules below. Search and training permissions are separate choices.

One path per line, such as /private/. Applies to all unblocked crawlers.

Your result

Enter your details or load an example, then run the tool.

What this tool can tell you

Review and merge with your existing file before publishing. Robots.txt is not access control and does not remove indexed pages.

Need evidence across your website? Explore CiteLadder’s site-wide workflow.