Learn · Answer
Should I block AI crawlers in robots.txt?
Short answer: Most businesses that want customers should allow the AI search bots and decide separately about the training bots. Search bots, such as OpenAI’s OAI-SearchBot, Anthropic’s Claude-SearchBot and PerplexityBot, decide whether you can be found and linked in AI answers. Training bots, such as GPTBot and ClaudeBot, collect content for model training. You can block one kind and allow the other.
Search bots: what blocking costs you
- OpenAI’s OAI-SearchBot is used to surface websites in ChatGPT’s search features; sites that opt out don’t appear in ChatGPT search answers, though they can still show as navigational links.[1]
- Anthropic’s Claude-SearchBot improves search results for Claude’s users; blocking it means your content isn’t indexed for search, which may reduce your visibility in Claude’s answers.[2]
- PerplexityBot surfaces and links websites in Perplexity’s search results and isn’t used to train foundation models.[3]
Training bots: your call
- OpenAI’s GPTBot crawls content that may be used to train its models. OpenAI says each setting is independent, so you can block GPTBot and still allow OAI-SearchBot.[1]
- Anthropic’s ClaudeBot collects content that could contribute to training; blocking it excludes your future content from training data.[2]
- Google-Extended is a robots.txt token, not a separate crawler. It controls use of your content for Gemini models and, Google says, doesn’t affect your inclusion or ranking in Google Search.[4]
Fetchers that act for a person
Some fetchers visit a page because a person asked about it. Perplexity says Perplexity-User generally ignores robots.txt for that reason.[3] OpenAI says ChatGPT-User isn’t used to decide what appears in search, and robots.txt rules may not apply to it.[1] robots.txt is a request, not a lock: the protocol itself says its rules are not a form of access authorization.[5]
A robots.txt that allows search and blocks training
Each group names a bot with User-agent and lists what it may not fetch with Disallow. A crawler follows the group that names it, and the “*” group only when no group does.[5] So to block GPTBot, ClaudeBot and Google-Extended while allowing everything else, give them a group of their own with “Disallow: /” and leave the search bots under your normal rules. A bot named in its own group ignores the “*” rules, so repeat any Disallow lines it should still follow. Test the file after every change.
Questions
How quickly does ChatGPT notice a robots.txt change?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.[1]
Does Anthropic follow robots.txt?
Anthropic says its bots honor robots.txt directives, and notes that IP blocking may not work reliably because it stops the bots reading the file at all.[2]
Related
Sources
- OpenAI: Overview of OpenAI crawlers, read on 6 November 2026
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, read on 6 November 2026
- Perplexity: Perplexity crawlers, read on 6 November 2026
- Google Search Central: Google’s common crawlers (Google-Extended), read on 6 November 2026
- RFC 9309: Robots Exclusion Protocol, read on 6 November 2026
Updated 6 November 2026. Something wrong or out of date? Email connect@odeist.com (subject: Correction: Should I block AI crawlers in robots.txt?).