Robots.txt generator

Choose which crawlers to allow, add your sitemap and download a robots.txt to start from.

Your input is processed in this browser. Text and files are not uploaded or saved by this tool. Use up to 200,000 characters per text area.

Your input

Block selected crawlers

Unchecked crawlers follow the general path rules below. Search and training permissions are separate choices.

One path per line, such as /private/. Applies to all unblocked crawlers.

Enter your details or load an example, then run the tool.

Your result

Enter your details or load an example, then run the tool.

What this tool can tell you

Review and merge with your existing file before publishing. Robots.txt is not access control and does not remove indexed pages.

Need evidence across your website? Site Health inspects captured page evidence before you choose a fix.

Explore Site Health