AI crawler
An AI crawler is a bot that fetches web pages for an AI company, either to train models or to answer people’s questions with search. The two jobs usually have different bots, and you can allow one and block the other.
OpenAI uses GPTBot to crawl content that may be used to train its models, and OAI-SearchBot to surface websites in ChatGPT’s search features; it says each setting is independent of the others.[1] Anthropic runs ClaudeBot for training, Claude-SearchBot for search and Claude-User when a person asks Claude about a page, and says its bots honor robots.txt.[2] Perplexity says PerplexityBot surfaces and links websites in its search results and is not used to train AI foundation models.[3]
Google-Extended is different: it is not a crawler but a robots.txt token that controls whether content Google crawls may be used for Gemini models, and Google says it does not affect a site’s inclusion or ranking in Google Search.[4]
Related
Sources
- OpenAI: Overview of OpenAI crawlers, read on 6 November 2026
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, read on 6 November 2026
- Perplexity: Perplexity crawlers, read on 6 November 2026
- Google Search Central: Google’s common crawlers (Google-Extended), read on 6 November 2026
Updated 6 November 2026. Something wrong or out of date? Email connect@odeist.com (subject: Correction: AI crawler).