Learn › Glossary

robots.txt

robots.txt is a plain text file at the top of a website that tells crawlers which paths they may and may not fetch.

The rules are defined in RFC 9309, the Robots Exclusion Protocol. They must sit in a file named /robots.txt, all lowercase, in the top-level path of the site.[1] Each group of rules names one or more crawlers with User-agent lines and lists the paths they may not fetch with Disallow lines.[1]

robots.txt is a request, not a lock. The RFC says the rules are not a form of access authorization.[1] Fetchers that act for a person can behave differently: Perplexity says its Perplexity-User fetcher generally ignores robots.txt because a user asked for the page.[2]

Related

Sources

  1. RFC 9309: Robots Exclusion Protocol, read on 6 November 2026
  2. Perplexity: Perplexity crawlers, read on 6 November 2026

Updated 6 November 2026. Something wrong or out of date? Email connect@odeist.com (subject: Correction: robots.txt).

Try Odeist AI free: 7 days, no card · Learn · Pricing