AI crawler checker

Every AI crawler we check

The user agents AI companies document for their crawlers, what each one feeds, and what blocking it means. Each row links to the operator's own page. Last checked 23 October 2026.

User agent	Operator	Purpose	robots.txt	If you block it
GPTBot	OpenAI	Model trainingOpenAI model training	Follows it	Your content should not be used to train OpenAI's models. ChatGPT search can still cite you. Source
OAI-SearchBot	OpenAI	AI search and citationsChatGPT search	Follows it	Your pages can drop out of ChatGPT's search answers and links. Source
ChatGPT-User	OpenAI	Live fetch for a userChatGPT, when a user asks it to open a page	May ignore it	OpenAI says robots.txt rules may not apply to these user-initiated visits. Source
ClaudeBot	Anthropic	Model trainingClaude model training	Follows it	Your future content should be excluded from Anthropic's training data. Source
Claude-SearchBot	Anthropic	AI search and citationsClaude's web search	Follows it	Anthropic says it may reduce your visibility in Claude's search results. Source
Claude-User	Anthropic	Live fetch for a userClaude, when a user's question needs a page	Follows it	Claude cannot fetch your pages to answer users, which may reduce your visibility. Source
PerplexityBot	Perplexity	AI search and citationsPerplexity search results	Follows it	Your pages can drop out of Perplexity's answers and source links. Perplexity says it is not used for model training. Source
Perplexity-User	Perplexity	Live fetch for a userPerplexity, when a user's question needs a page	May ignore it	Perplexity says this fetcher generally ignores robots.txt because a user asked for the page. Source
Google-Extended	Google	Control tokenGemini training and grounding	Follows it	Limits use of your content for Gemini training and grounding. Google says it does not affect inclusion in Google Search. Source
Googlebot	Google	AI search and citationsGoogle Search, including AI Overviews	Follows it	Your pages drop out of Google Search. Almost never what you want. Source
Bingbot	Microsoft	AI search and citationsBing search and Microsoft Copilot answers	Follows it	Your pages drop out of Bing, and with it the web results Copilot can use. Source
Applebot-Extended	Apple	Control tokenApple foundation model training	Follows it	Opts your content out of training Apple's generative models. Apple says it does not crawl and does not affect search. Source
Applebot	Apple	AI search and citationsSiri, Spotlight and Safari search	Follows it	Your pages can drop out of Siri and Spotlight suggestions. Source
meta-externalagent	Meta	Model trainingMeta AI model training	Follows it	Meta's crawler for training foundation AI models stops collecting your pages. Source
meta-webindexer	Meta	AI search and citationsMeta AI search results	Follows it	Meta says allowing it helps Meta AI cite and link your content. Source
Amazonbot	Amazon	Model trainingAmazon products, may train Amazon AI models	Follows it	Amazon's general crawler, which it says may be used to train its AI models, stops collecting your pages. Source
DuckAssistBot	DuckDuckGo	AI search and citationsDuckDuckGo's AI-assisted answers	Follows it	Your pages stop appearing as sources in DuckAssist answers. DuckDuckGo says the data is not used to train models. Source
CCBot	Common Crawl	Model trainingCommon Crawl's open web archive, widely used to train models	Follows it	Your pages stop entering the open Common Crawl dataset. Source

Following robots.txt is voluntary. Operators describe their own crawlers; we report what they say.