AI and search crawler directory
One page per crawler, covering what it is, the robots.txt rules that control it, how to verify its requests, and how the 10,000 most popular sites treat it.
AI training crawlers
Bots that collect web content to train AI models.
- Applebot-ExtendedApplebot-Extended is Apple's robots.txt token for opting out of AI model training. What blocking it changes, how to verify Applebot and what top sites do.
- ClaudeBotWhat ClaudeBot does, what blocking it changes according to Anthropic, how to verify real requests, and how the 10,000 most popular sites treat it in robots.txt.
- Google-ExtendedGoogle-Extended is a robots.txt token from Google covering Gemini training and grounding. Learn what blocking it changes, how to verify requests and what top sites do.
- GPTBotGPTBot is OpenAI's crawler for training data. This page covers what it does, what blocking it changes, how to verify requests, and how top sites handle it.
AI search crawlers
Bots that index pages so AI assistants can find and cite them in answers.
- Claude-SearchBotWhat Claude-SearchBot does, what blocking it changes according to Anthropic, how to verify real requests, and how the 10,000 most popular sites handle it in robots.txt.
- OAI-SearchBotWhat OAI-SearchBot does, what blocking it changes for ChatGPT search answers, how to verify its requests, and how the 10,000 most popular sites treat it in robots.txt.
- PerplexityBotPerplexityBot is Perplexity's search crawler. Learn what it is for, what blocking it changes, how to verify real requests, and how the top 10,000 sites handle it.
User-triggered AI fetchers
Agents that load a page when a person asks an AI assistant about it.
- ChatGPT-UserChatGPT-User is the agent OpenAI uses when ChatGPT visits a page for a user. What a robots.txt rule changes, how to verify requests, and what top sites do.
- Claude-UserLearn what Anthropic's Claude-User fetcher does, what blocking it changes, how to verify real requests, and how the 10,000 most popular sites treat it in robots.txt.
- Perplexity-UserPerplexity-User fetches pages when Perplexity users ask questions. Learn what it does, how robots.txt applies to it, how to verify requests, and what top sites do.
Search engine crawlers
Traditional search bots whose access also affects how AI features can use your content.
- ApplebotApplebot is Apple's web crawler for Spotlight, Siri and Safari. Learn what blocking it changes, how Applebot-Extended and nosnippet work, and how to verify requests.
- BingbotBingbot is Microsoft's search engine crawler. How to allow or block it in robots.txt, how to verify real requests, and what the 10,000 most popular sites do with it.
- GooglebotGooglebot is Google's search crawler. Here is what a block changes, how it relates to Google-Extended, how to verify real requests, and what the top sites do.