Skip to content
VYZZ LogoVYZZ

Free tool · AI crawler checker

Is your website blocking ChatGPT?

Enter your website. We read your robots.txt the way AI crawlers do, then request your homepage as ChatGPT, Claude and Perplexity to catch firewall blocks that robots.txt doesn't show.

Free. No sign-up. We read public files only and store nothing.

What we check

The crawlers behind each AI assistant.

Each company runs separate crawlers for answering questions and for training models. These are the names your robots.txt has to get right, from each company's own documentation.

AssistantCrawler (user agent)Used for
ChatGPTOAI-SearchBotAnswers and citations
ChatGPT-UserOpens pages people ask about
GPTBotModel training
ClaudeClaude-SearchBotAnswers and citations
Claude-UserOpens pages people ask about
ClaudeBotModel training
PerplexityPerplexityBotAnswers and citations
Perplexity-UserOpens pages people ask about
Gemini and Google AI OverviewsGooglebotGoogle Search, including AI Overviews and AI Mode
Google-ExtendedGemini training and Gemini app answers (a robots.txt token, not a crawler)
Microsoft CopilotBingbotAnswers and citations

The one decision

Block training if you want. Never block search.

Training crawlers collect pages to build future models. Search crawlers fetch your pages so an assistant can quote and link to you today. They are separate on purpose, so you can say no to one and yes to the other.

Keep open

OAI-SearchBot · ChatGPT-User · Claude-SearchBot · Claude-User · PerplexityBot · Perplexity-User · Googlebot · Google-Extended · Bingbot

Block these and the assistant can’t read your pages when a customer asks about you. It falls back to what other sites say about you, or names a competitor it can read.

Your call

GPTBot · ClaudeBot

Blocking these keeps your pages out of future training data. It doesn’t remove you from today’s answers, which come from the search crawlers.

Copy this

A robots.txt that lets AI in.

Replace the folders with your own private ones. Every group repeats them, because a crawler that finds its own group ignores the * group entirely.

# Let AI assistants find and cite this site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/
Disallow: /checkout/

# Everyone else
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/

Sitemap: https://yourbusiness.com/sitemap.xml

Access is the first of four things the free audit checks.

Then content, structured data and authority, with a score for each.

Run the free audit

FAQ

Questions people ask.

  • Does blocking GPTBot remove my business from ChatGPT?

    No. OpenAI uses separate crawlers: GPTBot collects pages for training, OAI-SearchBot fetches pages for ChatGPT's search answers, and ChatGPT-User opens a page when someone asks ChatGPT to. You can block GPTBot and keep OAI-SearchBot open. Block OAI-SearchBot and ChatGPT can't read your pages when it answers.

  • What is robots.txt?

    A plain text file at yourwebsite.com/robots.txt that tells crawlers which pages they may fetch. The rules are an internet standard, RFC 9309. A crawler follows the group that names it and ignores the general User-agent: * group, so one wrong group can shut out an assistant you meant to welcome.

  • My robots.txt allows everything, but the check says a crawler was refused. Why?

    Something in front of your website is turning it away: a firewall, your hosting provider or a security plugin. Cloudflare, for example, changed its default to block AI crawlers in July 2025. Look for a bot-blocking or AI-crawler setting in your firewall or CDN dashboard and allow the search crawlers.

  • Is Google-Extended a crawler? Does blocking it hurt my Google ranking?

    It isn't a crawler. It's a name you can use in robots.txt; Google's regular crawlers do the fetching. Google says blocking it doesn't affect inclusion or ranking in Google Search. It does decide whether your pages can be used to train Gemini and to ground answers in the Gemini app, so blocking it costs you there.

  • Do ChatGPT-User and Perplexity-User follow robots.txt?

    Not always. They fetch a page because a person asked for it, so OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it. The checker shows what your file says; those two may fetch the page anyway.

  • What happens if my robots.txt returns an error?

    If the file is missing (a 404), crawlers treat the whole site as open. If the server errors (a 5xx) or doesn't answer, the standard tells crawlers to assume everything is off limits until the file loads again. A broken robots.txt can hide a whole site from AI.

  • How quickly do changes take effect?

    RFC 9309 says crawlers shouldn't rely on a cached copy of robots.txt for more than 24 hours, so expect a day. Getting back into answers takes longer, because each assistant re-reads your pages on its own schedule.

  • Do you store my website or the results?

    No. The check reads your public robots.txt and homepage, shows you the result and keeps nothing. Run it as often as you like.

Sources and how we checked

Crawler names and purposes come from each company's crawler documentation, linked below and in the table above. robots.txt is evaluated under RFC 9309, the standard Google, OpenAI, Anthropic and others follow. Vendors add and rename crawlers; if something here is out of date, email hello@vyzz.io.

More free tools

When your customer asks ChatGPT… your competitor shows up.

SEO shaped the last decade. GEO will decide the next.

*Free AI visibility scan. No obligation. No credit card required.

Optimized for

  • ChatGPT
  • Claude
  • Gemini
  • Grok
  • Copilot
  • Perplexity