Is your website blocking the bot behind ChatGPT's answers?

The one file on your site you have probably never opened

Type your web address into a browser, add /robots.txt on the end, and press enter. A page of plain text loads. That file has been sitting on your website the whole time you have owned it, and almost every owner I talk to has never read a line of it.

The file works like a sign on the door for automated visitors. Some of those visitors are bots that read your pages so ChatGPT or Perplexity can quote you back to a customer who just asked a question. If your sign tells one of those bots to stay out, the assistant has nothing of yours to hand over, and the customer hears the name of the shop down the street instead.

Almost no owner chose this on purpose. In most cases a security plugin arrived with a "block AI scrapers" toggle already switched on, or whoever built the site added the rule and never mentioned it. You were never asked.

New research from cloro, published July 6, 2026, read the robots.txt file on 1,058 websites that AI assistants cite a lot. 13.9% of them shut out GPTBot, which gathers training data for OpenAI. OAI-SearchBot, the crawler OpenAI uses to surface websites in ChatGPT search, is shut out on only 3.4%. ClaudeBot, which gathers training data for Claude, is turned away by 15.4% of the sites. Google-Extended, which decides whether Google may train Gemini on your pages, by 13.2%.

Read those two numbers next to each other and something is backwards. Far more sites are closing the door on the training bot than on the bot that decides whether they get named at all.

How far the cloro numbers actually go

cloro also scored how often each site gets named by ChatGPT. The score counts how many times ChatGPT cites a site for every time that site turns up in a normal Google result, so a higher number means the site gets named more than its Google ranking alone would explain. Among sites that block GPTBot, the middle of the pack scored 0.003. Among sites that let GPTBot in, 0.417. Perplexity looks much the same. Sites blocking PerplexityBot scored 0, and the ones that allow it scored 1.167.

Now the part where you should slow down, because cloro does too. The study measures a link between two things and says plainly that it cannot show what causes what. Sites that turn away every bot in sight may also be small or new for reasons that have nothing to do with the file. The link in the numbers is strong all the same. Sites that shut out GPTBot are almost never named by ChatGPT.

The people who ran the study put it like this:

If being named in AI answers matters to you, a blanket "block every AI bot" robots.txt is working against you.

That is cloro's own summary of their findings, from July 6, 2026.

Two OpenAI bots, two separate switches

This is where the file gets confusing for owners. OpenAI runs more than one bot, and they do different jobs.

GPTBot reads your pages so OpenAI can train its models on them. The bot that matters for getting named is OAI-SearchBot, which crawls the web so ChatGPT search can show your site in its answers. A third one, ChatGPT-User, visits a page when a person's request in ChatGPT calls for it. Per OpenAI's own documentation, GPTBot and OAI-SearchBot are separate settings that you control on your own site, while robots.txt rules may not apply to ChatGPT-User because a person asked for the page. A change to your file takes about 24 hours to reach OpenAI's search systems.

Wanting your writing kept out of AI training is a fair thing to want. Plenty of owners feel that way, and it is their site and their call. The trouble is that two switches got treated as one. Most owners who blocked training were never told there was a second switch, and never meant to shut out the bot that brings customers.

The five-minute check anyone can run

A browser and five minutes will do this part, with no developer involved. It works the same way in every trade, since a dental practice and an HVAC shop have the same file sitting at the same kind of address.

1. Open the file. Type your web address, add /robots.txt on the end, press enter. If you get an error page, the file does not exist, which usually means nothing is being turned away.
2. Look for the bot names. GPTBot and Google-Extended feed training. OAI-SearchBot and PerplexityBot are the search crawlers that decide whether ChatGPT and Perplexity can show your pages. ChatGPT-User and Perplexity-User fetch a page when a person asks for it. Use your browser's find command to search the page for each one.
3. Read the line under each name. A line starting with User-agent names one bot. The lines below it say what that bot may read.
4. Watch for the catch-all. A User-agent: * line followed by Disallow: / shuts out every bot at once, including the ones that decide whether ChatGPT names you.
5. Send it on. Screenshot the page and send it to whoever built or hosts your site. Ask them why the bot behind AI answers is on that list, and who put it there.

Here is what a closed door looks like:

User-agent: GPTBot
Disallow: /

Two lines of text. That is all it takes to keep OpenAI's training bot off every page you own.

One more date worth writing down if your site runs behind Cloudflare, a service that sits in front of many websites to speed them up and filter traffic. On July 1, 2026, Cloudflare announced new defaults starting September 15, 2026: on new domains, pages that show ads turn away training bots and agent bots, the ones that fetch pages for a person, unless the owner changes the setting. Search bots stay allowed. That came a year after Cloudflare switched its default to block AI crawlers on July 1, 2025. If a site of yours went live on Cloudflare in the last year, open its AI crawler settings and check what they allow.

What to do with what you find

If your file comes back clean, you have spent five minutes and you can stop thinking about this one. Plenty of sites are clean.

If GPTBot is blocked and you never asked for that, the call is yours to make. Keeping OpenAI out of training data is a defensible choice, and you can hold that line while still letting OAI-SearchBot crawl the pages ChatGPT search can show. Whoever set your site up can make that edit in a few minutes. Then give it about a day to reach OpenAI.

If everything is shut out by a catch-all line, ask why before you touch a thing. Sometimes there is a real reason behind it, like a staging site or a bot problem that got out of hand last year. In plenty of cases, though, a plugin default was switched on and nobody went back to read what it did.

A medspa owner told me last month that she had always assumed her site was open to everything, because why would it be anything else. That assumption is the whole problem. The file is 30 seconds away, and it answers the question for you in plain text.

Corrected October 12, 2026: an earlier version said OAI-SearchBot fetches pages while ChatGPT writes an answer and listed PerplexityBot as a live fetcher. Per OpenAI and Perplexity, those two are search crawlers, and ChatGPT-User and Perplexity-User do the live fetching. We also added links to Cloudflare's announcements.