Your own website may be blocking AI from finding you

Two readers show up when AI answers about you

You can do everything right on your Google profile and your website and still be missing from AI answers. The cause usually hides on your own site. A setting you never touched can turn away the automated reader an AI assistant sends to fetch your page before it answers a customer. It is one of the most common things we find on sites that are missing from AI answers, and the owner almost never set it on purpose. This post shows you how to find that setting and open it, in about five minutes.

Start with what happens when a customer asks. When someone asks ChatGPT, Perplexity, or Google's AI about a business like yours, the answer is often built on the spot. The assistant grabs a few live web pages in that moment and quotes from them. To grab a page, it sends out a small piece of software, sometimes called a crawler or a bot. Think of it as an automated reader an AI search engine sends to fetch your page.

Here is the part most owners miss. There are two kinds of these readers, and they do different jobs.

The first kind trains the models. These are the training readers, with names like GPTBot, Google-Extended, and ClaudeBot. They pull pages to help teach the AI over time. Blocking them is a fair choice, and it doesn't cost you a spot in today's answers.

The second kind fetches your page at the moment of a question so the assistant can name you. These are the answer readers, with names like OAI-SearchBot, ChatGPT-User, PerplexityBot, and the reader Claude uses to fetch a page. OpenAI runs both kinds and lists them in its bot documentation. For getting named in an answer, the answer readers are the ones that count. Block them, and you drop out of what the assistant says back.

That difference is the whole game today. A lot of owners think blocking GPTBot pulls them out of ChatGPT answers. In fact GPTBot is only a training reader, so blocking it leaves your spot in the answers untouched. The real risk is a blanket block that also turns away the answer reader. That is the one that quietly costs you customers.

How your own site ends up blocking them

Most owners never chose to block anything. The block came switched on, from a tool they set up for a different reason.

Start with Cloudflare. Cloudflare is a service that sits in front of your website to speed it up and filter out bad traffic. Millions of businesses run it. Since July 1, 2025, every new site added to Cloudflare has shipped with the common AI readers blocked by default. Nobody asks the owner. The block is just on. Through 2026, people who track this traced it to one default rule that turns away known AI readers on any new site.

Cloudflare widened that default again this summer. In a July 1, 2026 announcement, it set a new default that starts September 15, 2026. From that date, more sites, mainly ones that show ads, block the training readers by default. The readers that answer live questions stay allowed, so that newer default keeps the answer readers in. That is the good news. The catch is that you should still check your own site rather than assume it, because the older default and other tools aren't so careful.

WordPress security plugins are the other common source. Several popular ones carry an option to block AI bots, and it is easy to switch on by accident. When it is on, it can catch the answer reader along with everything else.

There's one more trap worth knowing. A block can live in two separate places. One is robots.txt, a small text file on your website that tells automated visitors what they can read, reachable by typing your web address followed by /robots.txt. The other is the security service or plugin in front of your site. These two can disagree. Your text file can say welcome while the traffic filter says no, and the traffic filter wins. So a real check has to cover both.

This is more common than it sounds. In a July 2026 crawl of the web by US Tech Automations, about 19% of sampled sites blocked at least one AI reader, and about 84% had set no AI policy at all. That second number is the one to sit with. About 84% of sites have never set any AI-reader policy at all, which means that for most of them the block came from a default setting that shipped switched on.

The five-minute check

Here is the whole task, and it takes about five minutes.

1. Open a browser and type your web address followed by /robots.txt. Read what comes up. You are looking for lines that name the answer readers, like OAI-SearchBot, ChatGPT-User, or PerplexityBot, next to the word Disallow. If you see them blocked, that is a problem to fix.
2. Log into your security service or plugin. In Cloudflare, look under Security, then Bots, then the AI scrapers and crawlers setting. In a WordPress plugin, look for an AI bots or bad bots option.
3. Set the answer readers to allowed. If the tool offers only a broad on or off switch for AI bots, use the finer setting that handles training and answer readers apart, so you can keep the answer readers in.
4. Save your change, then load your robots.txt one more time so both places agree.

If none of that is blocked, you're already in good shape, and the check still took five minutes. If it was blocked, you just reopened the door to the reader that gets you named.

What this means for the next few months

AI answers are picky to begin with. A 2026 study by SOCi found that assistants recommend only a small share of local businesses, far fewer than Google's map results do. When the field is that narrow, handing the one reader that could name you a locked door is an expensive mistake to make by accident.

None of this replaces the rest of your work. A strong profile and real prices on your pages still matter. But all of that assumes an AI can reach your page in the first place. Access comes first. If the answer reader cannot get in, the best profile in your market never gets read.

So put five minutes on the calendar this week. Type your web address followed by /robots.txt, read it, check your security setting, and make sure the answer readers can get in. It is your site and your call, and today it is a call worth making.