robots.txt
robots.txt is a plain text file at the top of a website that tells crawlers which paths they may and may not fetch.
The rules are defined in RFC 9309, the Robots Exclusion Protocol. They must sit in a file named /robots.txt, all lowercase, in the top-level path of the site.[1] Each group of rules names one or more crawlers with User-agent lines and lists the paths they may not fetch with Disallow lines.[1]
robots.txt is a request, not a lock. The RFC says the rules are not a form of access authorization.[1] Fetchers that act for a person can behave differently: Perplexity says its Perplexity-User fetcher generally ignores robots.txt because a user asked for the page.[2]
Related
Sources
- RFC 9309: Robots Exclusion Protocol, read on 6 November 2026
- Perplexity: Perplexity crawlers, read on 6 November 2026
Updated 6 November 2026. Something wrong or out of date? Email connect@odeist.com (subject: Correction: robots.txt).