Begin with the crawler purpose
A single provider can operate several agents. Blocking the training agent does not necessarily block search discovery, and allowing a search bot does not opt a site into every other use.
Current agent map
Names and provider policies can change. Treat the provider documentation linked below as the source of truth and review policy after major product changes.
| Provider | Agent or token | Primary purpose | Practical meaning |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Search discovery | Used to surface sites in ChatGPT search. |
| OpenAI | GPTBot | Model training | Separate from search discovery; can be controlled independently. |
| OpenAI | ChatGPT-User | User-triggered fetch | Acts for a user request and may not follow robots.txt in the same way as automated crawling. |
| Perplexity | PerplexityBot | Search discovery | Indexes public pages for Perplexity search results. |
| Perplexity | Perplexity-User | User-triggered fetch | Fetches a page when a user asks Perplexity to access it. |
| Anthropic | Claude-SearchBot | Search discovery | Supports search and answer discovery. |
| Anthropic | ClaudeBot | Model training | Used for model-development crawling and controlled separately. |
| Anthropic | Claude-User | User-triggered fetch | Retrieves content in response to a direct Claude user action. |
| Googlebot | Search discovery | Controls crawling for Google Search, which underpins Google AI search experiences. | |
| Google-Extended | Generative AI control | A separate control token; it does not control Google Search inclusion or ranking. | |
| Microsoft | bingbot | Search discovery | Crawls for Bing search surfaces used by Microsoft search experiences. |
Choose policy by business intent
Decide which public content should be discoverable in AI-assisted search, which content can be fetched only through direct user actions, and whether model-training access aligns with the organization policy.
- Keep private, licensed, account-only, and sensitive content behind authentication. robots.txt is not access control.
- Allow desired search agents to public pages that should appear in answer and search surfaces.
- Control training agents separately instead of using one broad wildcard block without reviewing the consequence.
- Apply path rules to stable URL groups and test the exact canonical URLs that matter.
- Document the decision owner, review date, and reason for each provider-specific policy.
robots.txt examples
This example allows search discovery for OpenAI, Perplexity, and Anthropic while blocking their documented training agents. Adjust it to the current provider documentation and your policy.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /For a path-specific restriction, use the real public route rather than a vague keyword pattern. The most specific matching Allow or Disallow rule normally wins for agents following the standard matching behavior.
User-agent: OAI-SearchBot
Allow: /guides/
Disallow: /internal-research/robots.txt is not the same as firewall access
A page can say Allow in robots.txt and still return 401, 403, 429, a JavaScript challenge, or a blank shell to the crawler.
Review CDN bot rules, web application firewall policy, rate limits, country restrictions, and origin authentication. A user-agent string can be spoofed, so use provider verification guidance or published network information where available rather than trusting the header alone.
Verify policy after deployment
- 1Fetch robots.txt publiclyConfirm the final production file is reachable and not cached from an older deployment.
- 2Test exact pathsEvaluate the canonical URL and important directories for each desired agent.
- 3Inspect response statusCheck that the real page returns useful HTML, not only that robots.txt says Allow.
- 4Review server logsLook for expected agents, status codes, challenge responses, and unexpected volume.
- 5Schedule policy reviewRevisit provider documentation and business intent at a defined interval.
Official source documentation
Crawler policy questions
Should a site allow AI search bots but block training bots?
That is a valid policy choice. Search discovery, user-triggered retrieval, and training are different purposes. Configure each documented agent explicitly and confirm the result in server logs.
Does Allow in robots.txt guarantee that an AI crawler can fetch the page?
No. CDN rules, web application firewalls, authentication, rate limits, and origin errors can still return a block or challenge response.
Can robots.txt prevent every user-triggered AI fetch?
Not necessarily. Some user-triggered agents act on an explicit user request and may not treat robots.txt like an automated indexing crawler. Consult each provider policy and use access controls when content must remain private.