Can ChatGPT, Claude, Perplexity and Google see your website? Check which search engines and AI crawlers your robots.txt allows, and whether your page can appear in AI answers.
Rules are evaluated the way crawlers do (RFC 9309): the most specific matching rule wins. No login required.
AI assistants read websites through their own crawlers, and your robots.txt file decides which crawlers are allowed in. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training; Anthropic uses Claude-SearchBot and ClaudeBot; Perplexity uses PerplexityBot. If robots.txt blocks a search or AI search crawler, that service cannot find or cite your page. This checker reads your robots.txt and tests your page against 18 crawlers.
It depends on your goals. Blocking AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from AI answers, which most businesses want to appear in. Blocking only training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of model training while staying visible in search and AI search.
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search and AI Overviews use Googlebot, so blocking Google-Extended does not affect your search rankings.
Add "User-agent: GPTBot" followed by "Disallow: /" to robots.txt, and do not block OAI-SearchBot or ChatGPT-User. ChatGPT search will still be able to find and cite your pages.
llms.txt is an emerging, optional file at your site root that gives AI assistants a short, plain-text summary of your business and links to your key pages. It does not control access — robots.txt does that.
The major search engines and AI companies listed here say they follow robots.txt. Some scrapers ignore it, so robots.txt is a request, not a security measure — never use it to hide private content.
Last updated: · Free tool by Regdec Enterprise
Free Web Tools