AI crawlers and robots.txt: which bots do what, and what to allow
OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended and more — what each AI crawler does, and how to set robots.txt rules deliberately.
Why “block AI bots” is the wrong question
It has become common to add a block of AI user agents to robots.txt in one go. The trouble is that these crawlers do very different things. Blocking a training crawler is a decision about whether your content may be used to build future models. Blocking a search crawler can remove your pages from an AI product’s answers — which, if you want to be visible there, is the opposite of what you intended.
The table below summarises what vendors document as of October 2026. Crawler names and behaviour change, so check the linked documentation before you act.
The main crawlers and what they do
| Token | Operator | Purpose (per vendor docs) | robots.txt |
|---|---|---|---|
OAI-SearchBot |
OpenAI | Surfaces websites in ChatGPT search features | Respected |
GPTBot |
OpenAI | Crawls content that may be used to train OpenAI’s foundation models | Respected |
ChatGPT-User |
OpenAI | Fetches pages when a user’s request in ChatGPT or a GPT needs them | May not apply (user-initiated) |
PerplexityBot |
Perplexity | Surfaces and links websites in Perplexity search results; not for training | Respected |
Perplexity-User |
Perplexity | Fetches a page to answer a specific user question | Generally not applied (user-initiated) |
Googlebot |
Crawls for Google Search, including AI features in Search | Respected | |
Google-Extended |
A control token (not a separate crawler) for whether content Google crawls may be used for Gemini training and grounding; does not affect Google Search inclusion | Respected | |
Bingbot |
Microsoft | Crawls for Bing, whose index also supports Microsoft Copilot experiences | Respected |
Sources: OpenAI crawler documentation, Perplexity crawler documentation, Google common crawlers. Other operators, including Anthropic, Apple and Common Crawl, publish their own tokens — look up each one rather than copying lists from blog posts.
Three common policies
1. Maximum discoverability. Allow everything. This is the default if robots.txt doesn’t mention a bot.
User-agent: *
Allow: /2. Visible in AI search, opted out of training. Allow search crawlers; disallow training-specific crawlers and tokens.
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Note that disallowing Google-Extended also opts out of Gemini grounding, not only training — weigh that if Gemini visibility matters to you.
3. Selective. Allow public marketing content, disallow areas such as internal search results, account pages or paid content for all bots.
Things robots.txt can’t do
- It is a request, not an access control. Well-behaved crawlers follow it; anything that must be private needs authentication.
- User-initiated fetchers may not follow it. OpenAI and Perplexity both document that user-triggered fetches may not apply robots.txt rules.
- It doesn’t remove what was already collected. Changes affect future crawling; vendors note updates can take around a day to be picked up.
- Your CDN may override it. Firewall “bot fight” settings can block crawlers you allowed in robots.txt. Check your CDN’s bot rules and logs.
What we do on prisol.org
We publish a public, explanatory robots.txt that allows search engines and AI search crawlers to access all public pages, because being discoverable in AI answers is the point of our work. Our decision on training crawlers is documented in the file itself. We also publish an llms.txt summary of the site — an emerging convention that may help some tools, but which no major platform has said it uses for ranking or inclusion.
If you’re not sure what your current setup allows, a technical SEO review will tell you — including what your CDN is doing.