Is your website blocking the bot behind ChatGPT's answers?
A plugin or your web host may have shut ChatGPT's bot out of your site without asking. Open your robots.txt file and read what it says.
Co-Founder, Vyzz
The one file on your site you have probably never opened
Type your web address into a browser, add /robots.txt on the end, and press enter. A page of plain text loads. That file has been sitting on your website the whole time you have owned it, and almost every owner I talk to has never read a line of it.
The file works like a sign on the door for automated visitors. Some of those visitors are bots that read your pages so ChatGPT or Perplexity can quote you back to a customer who just asked a question. If your sign tells one of those bots to stay out, the assistant has nothing of yours to hand over, and the customer hears the name of the shop down the street instead.
Almost no owner chose this on purpose. In most cases a security plugin arrived with a "block AI scrapers" toggle already switched on, or whoever built the site added the rule and never mentioned it. You were never asked.
New research from cloro, published July 6, 2026, read the robots.txt file on 1,058 websites that AI assistants cite a lot. 13.9% of them shut out GPTBot, which gathers training data for OpenAI. The bot that actually fetches pages while ChatGPT writes an answer, OAI-SearchBot, is shut out on only 3.4%. ClaudeBot, which gathers training data for Claude, is turned away by 15.4% of the sites. Google-Extended, which decides whether Google may train Gemini on your pages, by 13.2%.
Read those two numbers next to each other and something is backwards. Far more sites are closing the door on the training bot than on the bot that decides whether they get named at all.
How far the cloro numbers actually go
cloro also scored how often each site gets named by ChatGPT. The score counts how many times ChatGPT cites a site for every time that site turns up in a normal Google result, so a higher number means the site gets named more than its Google ranking alone would explain. Among sites that block GPTBot, the middle of the pack scored 0.003. Among sites that let GPTBot in, 0.417. Perplexity looks much the same. Sites blocking PerplexityBot scored 0, and the ones that allow it scored 1.167.
Now the part where you should slow down, because cloro does too. The study measures a link between two things and says plainly that it cannot show what causes what. Sites that turn away every bot in sight may also be small or new for reasons that have nothing to do with the file. The link in the numbers is strong all the same. Sites that shut out GPTBot are almost never named by ChatGPT.
The people who ran the study put it like this:
If being named in AI answers matters to you, a blanket "block every AI bot" robots.txt is working against you.
That is cloro's own summary of their findings, from July 6, 2026.
Two OpenAI bots, two separate switches
This is where the file gets confusing for owners. OpenAI runs more than one bot, and they do different jobs.
GPTBot reads your pages so OpenAI can train its models on them. The bot that matters for getting named is OAI-SearchBot, which goes out and fetches pages while ChatGPT is writing an answer with sources. A third one, ChatGPT-User, only shows up when a person asks ChatGPT to go open a specific page for them. Per OpenAI's own documentation, each of these is a separate setting that you control on your own site. A change to your file takes about 24 hours to reach OpenAI.
Wanting your writing kept out of AI training is a fair thing to want. Plenty of owners feel that way, and it is their site and their call. The trouble is that two switches got treated as one. Most owners who blocked training were never told there was a second switch, and never meant to shut out the bot that brings customers.
The five-minute check anyone can run
A browser and five minutes will do this part, with no developer involved. It works the same way in every trade, since a dental practice and an HVAC shop have the same file sitting at the same kind of address.
- Open the file. Type your web address, add /robots.txt on the end, press enter. If you get an error page, the file does not exist, which usually means nothing is being turned away.
- Look for the bot names. GPTBot and Google-Extended feed training. OAI-SearchBot, ChatGPT-User and PerplexityBot fetch pages to build live answers. Use your browser's find command to search the page for each one.
- Read the line under each name. A line starting with User-agent names one bot. The lines below it say what that bot may read.
- Watch for the catch-all. A
User-agent: *line followed byDisallow: /shuts out every bot at once, including the ones that decide whether ChatGPT names you. - Send it on. Screenshot the page and send it to whoever built or hosts your site. Ask them why the bot behind AI answers is on that list, and who put it there.
Here is what a closed door looks like:
User-agent: GPTBot
Disallow: /Two lines of text. That is all it takes to keep OpenAI's training bot off every page you own.
One more date worth writing down. Cloudflare, the service more than one in five websites run behind, said on July 1, 2026 that from September 15 it will turn away training and chat-agent bots by default on new sites that carry ads. Search bots stay allowed. That is the company's own announcement, so take it as their word, and give your settings a look if a new site of yours goes live after that date.
What to do with what you find
If your file comes back clean, you have spent five minutes and you can stop thinking about this one. Plenty of sites are clean.
If GPTBot is blocked and you never asked for that, the call is yours to make. Keeping OpenAI out of training data is a defensible choice, and you can hold that line while still letting OAI-SearchBot fetch the page ChatGPT quotes. Whoever set your site up can make that edit in a few minutes. Then give it about a day to reach OpenAI.
If everything is shut out by a catch-all line, ask why before you touch a thing. Sometimes there is a real reason behind it, like a staging site or a bot problem that got out of hand last year. In plenty of cases, though, a plugin default was switched on and nobody went back to read what it did.
A medspa owner told me last month that she had always assumed her site was open to everything, because why would it be anything else. That assumption is the whole problem. The file is 30 seconds away, and it answers the question for you in plain text.
Frequently asked questions
- What is robots.txt and where do I find mine?
- It is a small text file that sits at the root of your website. Type your web address, add /robots.txt on the end, and press enter to read it. It works like a sign on the door for automated visitors, telling each one which pages it may read. If your browser shows an error page instead, the file does not exist on your site.
- Can I keep my work out of AI training and still get named by ChatGPT?
- Yes. OpenAI runs GPTBot for training and OAI-SearchBot for the pages ChatGPT quotes in an answer, and they are separate settings you control. A site can turn GPTBot away and still let OAI-SearchBot through. A change to the file takes about 24 hours to reach OpenAI.
- Does blocking a bot definitely cost me mentions in AI answers?
- The honest answer is that nobody has proven that. cloro's July 2026 study found a strong link between the two and says outright that it cannot show what causes what. What the data does show is that sites blocking GPTBot are almost never named by ChatGPT, so it is worth knowing which side of that line you are on.