Skip to content
New guide: How to measure AI visibility — and what you can’t measure. Read it →
PRISOL

Free tool · runs in your browser

AI Robots.txt Generator — decide which AI crawlers can use your site.

Choose a policy or set each crawler yourself, add the paths you want kept out, and copy a robots.txt that is visible in AI search on your terms. Nothing you enter leaves your browser.

Start from a policy

“Block AI crawlers” keeps Googlebot, Bingbot and Applebot allowed so you stay in normal search results.

Search / answers
  • Allowed
  • Allowed
  • Allowed
  • Allowed
  • Allowed
  • Allowed
Model training
  • Allowed
  • Allowed
  • Allowed
  • Allowed
Control token (no crawling)
  • Allowed
  • Allowed
User-requested fetch
  • Allowed
  • Allowed
  • Allowed

Your robots.txt

How to choose an AI crawler policy

Each AI company runs crawlers with different jobs, and robots.txt lets you treat them differently. For a business that wants to be found, the decision usually comes down to two questions:

  • Do you want to appear in AI answers? Then allow the search crawlers — OAI-SearchBot (ChatGPT), PerplexityBot, Claude-SearchBot — plus Googlebot, Bingbot and Applebot for classic search and the AI features built on it.
  • Do you want your content used to train models? If not, block the training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot) and the control tokens Google-Extended and Applebot-Extended. Note that Google-Extended also covers Gemini grounding.

After you upload it

  1. Upload the file to the root of your domain as robots.txt.
  2. Check that your CDN or firewall isn’t blocking the crawlers you allowed — run the AI Crawler Checker.
  3. Expect changes to take up to a day: crawlers cache robots.txt.

Allowing crawlers makes your pages eligible; it doesn’t guarantee they will be cited. That depends on your content and reputation — see answer engine optimization and generative engine optimization.

FAQ

AI Robots.txt Generator: frequently asked questions

Can I stay visible in AI search but opt out of AI training?
Yes, for the operators that separate the two. Allow search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, and disallow training crawlers and tokens such as GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended. The “AI search, no training” preset does exactly this.
Does blocking Google-Extended affect my Google rankings?
Google says Google-Extended does not affect inclusion in Google Search. It does control whether content Google crawls can be used for Gemini training and grounding, so blocking it may reduce how Gemini uses your pages.
Will robots.txt stop every AI crawler?
No. robots.txt is a public request that reputable crawlers follow. User-initiated fetchers such as ChatGPT-User and Perplexity-User may not apply it, and it is not an access control: anything private needs authentication.
Where do I put the file?
Upload it as robots.txt at the root of your domain (for example https://example.com/robots.txt). Each subdomain needs its own file. Crawlers may cache it for up to a day, so changes are not instant.

Related guides and services

AI Visibility Audit

Is your brand visible where your customers are searching?

Our AI Visibility Audit shows how you appear across Google and AI answer engines today — and what to fix first.

Prefer email? prithvi@prisol.org