The AI crawler access problem most SMB sites have not audited

Check crawl access before anything else

AI companies run separate bots for training and for search. Blocking GPTBot keeps your pages out of OpenAI's model training, while ChatGPT search relies on a different bot, OAI-SearchBot. Blocking the search bots is what costs a site its citations. That distinction is the thing most SMB operators do not yet know exists, and a snippet pasted into robots.txt from a privacy blog years ago can be blocking the wrong ones.

Crawl access is the threshold question every audit needs to start with. Before an AI engine can cite a business, the engine has to be able to crawl the business. The signals everyone has spent four months tuning, schema, headers, FAQ blocks, earned media, freshness windows, become irrelevant when the bot cannot reach the page in the first place.

What crawl access is worth

Duda published a study in April 2026 covering 850,000 SMB sites and 69 million AI crawler visits. The headline finding: AI-crawled sites generated 320 percent more human traffic, 270 percent more form submissions, and 250 percent more click-to-call events than non-crawled peers in the same study cohort.

The lift the study measures is human traffic, downstream of AI citation. The full sequence has four stages running in order: an AI crawler fetches the page, the engine decides whether to cite the page in its answer, the citation appears in a user's ChatGPT or Google AI Overview response, and a fraction of those users click through to the SMB site. Every stage depends on the first one. Without crawl access, none of the later stages can occur.

The Duda data also identifies the operator-controllable factors that drive crawl rate. AI crawler visits rise by roughly 7 percent for every blog post added and roughly 4 percent for every additional indexable page, with local schema markup and Google Business Profile synchronization rounding out the top non-cadence drivers. None of this is exotic, and none of it is new advice. The novelty is that the same content cadence and schema work that drives Google ranking now drives a separate, parallel pipeline of AI traffic.

The robots.txt audit problem

TechnologyChecker.io analyzed robots.txt data that Cloudflare collects from the top 10,000 domains on the web. These are large, high-traffic sites, so read the numbers as a signal for the wider web rather than a count of small businesses. On March 30, 2026, 4,047 files were parsed. GPTBot was named in 13.8 percent of them, ClaudeBot in 11.5 percent, CCBot in 11.2 percent, and Google-Extended in 10.7 percent. Counting only files that block a bot from the whole site, GPTBot was at 8.9 percent and ClaudeBot at 8.0 percent.

ClaudeBot's share of the AI-bot Disallow rules in that data grew from 9.6 percent in early January 2026 to 10.1 percent by late March, the fastest rise of any bot in the period. While some operators block bots intentionally to protect training data, many others do so accidentally through privacy-blog templates, hosting provider defaults, or one-line copy-pastes during site setup that nobody has audited since.

The accidental-block problem is the size of an entire vertical's worth of SMB visibility. A single Disallow rule in robots.txt can erase a year of content work, schema work, and Google Business Profile work from the AI citation pool. The work is not lost in any other channel. Major search engines like Google and Bing continue to read the site normally, customers landing on the site from any other source still convert, and the AI search citation pipe is the one channel where a robots.txt rule is dispositive.

Training bots and search-citation bots are not the same agent

This is the technical detail that catches almost every operator off guard. The user agent that powers AI search citation is a different identifier from the one that fetches pages for training data. OpenAI uses GPTBot to gather training data and OAI-SearchBot to power its search citations. Anthropic splits its bots the same way: ClaudeBot collects training data, Claude-SearchBot indexes pages for Claude's search results, and Claude-User fetches a page when a Claude user's question calls for it. PerplexityBot surfaces and links sites in Perplexity's search results, and Perplexity says it isn't used to train AI models. OpenAI and Perplexity also run fetchers that open a page when a person asks for it, ChatGPT-User and Perplexity-User. CCBot is Common Crawl, a non-AI archive that AI training pipelines often consume.

An SMB that wants to opt out of training while staying visible in AI search citation can keep GPTBot and ClaudeBot disallowed and explicitly Allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Leave the person-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) alone too. OpenAI and Perplexity both say robots.txt rules may not apply to those anyway, since a person asked for the page. A blanket User-agent: * with Disallow: / cuts off both pipes at once. The privacy advice that propagates through tech blogs almost never makes this distinction, which is how operators end up blocking the bots that would have cited them.

The four-step audit any operator can run today

The audit takes about thirty minutes and does not require a developer.

1. Pull your robots.txt. Open yoursite.com/robots.txt in any browser. Read every Disallow rule. Note which AI user agents are named.

2. Decide your stance per user agent. Operators who want to prevent training-data scraping while keeping search citation can disallow GPTBot and ClaudeBot, then explicitly allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. All three are search bots, and all three almost always belong in the allowed set.

3. Check the Disallow-slash trap. A User-agent: * block followed by Disallow: / overrides every Allow rule below it for unmatched bots. If your file has this pattern, every AI search-citation bot is blocked unless explicitly named with its own Allow.

4. Check who actually visits. OpenAI, Anthropic, and Perplexity each document their bots' names; OpenAI's list is at developers.openai.com/api/docs/bots. Those pages explain what each bot does. They don't show who visited your site. For that, ask your web host or developer for the server access logs and search them for OAI-SearchBot, Claude-SearchBot, and PerplexityBot. If a search bot hasn't fetched a page in the last 30 days, check robots.txt and any firewall or bot-blocking setting at your host. OpenAI also asks sites to let through traffic from its published search-bot IP addresses.

For operators who want the full picture in one report, including which AI engines are citing your business today and where each citation traces back, an audit across ChatGPT, Claude, Perplexity, and Google AI Overviews shows the gap directly.

Frequently asked

How do I check if my site is blocking AI bots?

Open yoursite.com/robots.txt in any browser. Look for User-agent lines naming GPTBot, ClaudeBot, Claude-SearchBot, CCBot, OAI-SearchBot, PerplexityBot, or Google-Extended. A Disallow rule under any of these names is a block. A blanket User-agent: * with Disallow: / blocks every crawler, AI bots included.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot fetches training data while OAI-SearchBot powers the search-citation answers used in ChatGPT product responses. Blocking OAI-SearchBot eliminates search citations, a far more disruptive outcome than the training-data focus that disallowing GPTBot affects.

How fast does fixing a robots.txt block move citations?

OpenAI says its search systems pick up a robots.txt change in about 24 hours. How soon citations follow depends on the engine and the page. Check your access logs a few days after the fix to confirm the search bots are fetching pages again.

Corrected October 12, 2026: an earlier version credited the robots.txt figures to Cloudflare and described them as SMB sites. They come from TechnologyChecker.io's analysis of Cloudflare data on the top 10,000 domains. We also corrected a description of OpenAI's bot documentation as a test page, removed an undocumented Bing user agent, added Anthropic's Claude-SearchBot and Claude-User, and removed an opening client example we could not support.