AI Crawler Checker — can AI search engines reach your site?

What the AI Crawler Checker tests

AI search products can only use pages their crawlers can reach. This checker looks at the three places access is usually decided:

1. robots.txt — we read your file and apply the same matching rules crawlers use (RFC 9309): the group for the specific crawler if one exists, otherwise the * group, with the longest matching rule winning.
2. Page directives — X-Robots-Tag headers and <meta name="robots"> tags such as noindex, which keep a page out of search indexes even when crawling is allowed.
3. Your firewall or CDN — we request the page with each AI crawler’s user agent and compare the response with a neutral request. Bot-protection settings often block crawlers that robots.txt allows.

Search crawlers, training crawlers and user fetchers

AI companies run several crawlers with different jobs. Search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot decide whether your pages can appear and be cited in AI answers. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for future models. User fetchers such as ChatGPT-User retrieve a page when someone asks about it. Blocking training while allowing search is a common, legitimate choice — blocking search crawlers usually is not what a business that wants to be visible intends.

What to do if a crawler is blocked

• If robots.txt blocks a search crawler you want, edit its group — our AI robots.txt generator builds the rules for you.
• If the firewall test shows refusals, review your CDN’s bot management (for example “block AI bots” or “bot fight” settings) and allow the crawlers you want.
• If a page has noindex, remove it from pages you want found — and keep it on pages you don’t.

Crawler names and behaviour change. Each result links to the operator’s own documentation. For the full picture, read our guide to AI crawlers and robots.txt.