If you want to be cited by AI search, allow the search crawlers: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude, and Googlebot for Google’s AI Overviews and AI Mode. Whether to allow the training crawlers, GPTBot, ClaudeBot and the Google-Extended control, is a separate decision that does not affect those search features. Most sites that are invisible in AI answers are not blocked on purpose; a plugin, a firewall rule or an old robots.txt line is doing it.
The crawlers that matter, and what each does
| Token | Company | Purpose | Block it and… |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search answers | You will not appear in ChatGPT search answers, except as a navigational link |
| GPTBot | OpenAI | Collects content that may train OpenAI models | Your content is excluded from training; ChatGPT search is unaffected |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to | OpenAI says robots.txt may not apply to these user-initiated visits |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; not used for training | You drop out of Perplexity’s results |
| Perplexity-User | Perplexity | Fetches pages for a user’s question | Perplexity says it generally ignores robots.txt because a user asked |
| Googlebot | Search, including AI Overviews and AI Mode | You disappear from Google Search entirely | |
| Google-Extended | A control token for Gemini training and grounding outside Search | No effect on Search inclusion or ranking | |
| ClaudeBot | Anthropic | Collects content that may train Anthropic models | Future content excluded from training |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude’s search results | Lower visibility in Claude’s search answers |
| Claude-User | Anthropic | Fetches pages when a Claude user asks | Claude cannot retrieve your page for users |
Two facts from the official documentation are widely misunderstood. First, Google says AI is built into Search and that robots.txt rules for Googlebot are the control for AI Overviews and AI Mode; Google-Extended does not remove you from them. Second, OpenAI treats search and training separately: you can allow OAI-SearchBot to appear in ChatGPT answers while disallowing GPTBot.
A sensible robots.txt for a business that wants AI visibility
This version allows search and answer crawlers, allows training crawlers too (remove those groups if you prefer to opt out of training), and keeps private areas closed:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart
Disallow: /checkout
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://www.example.com/sitemap.xmlA crawler that matches a specific group follows only that group, so repeat any disallow rules you need inside it. To opt out of training while staying in AI search, change the GPTBot and ClaudeBot groups to Disallow: / and add a Google-Extended group with Disallow: /.
Where blocks hide
- Security and SEO plugins on WordPress that add “block AI bots” rules without much warning.
- CDN and firewall settings. Cloudflare and similar services offer one-click AI crawler blocking, and some accounts have it switched on by default. OpenAI and Perplexity both note that firewall rules returning 403 errors stop their crawlers even when robots.txt allows them.
- Shopify’s robots.txt template. Shopify generates robots.txt from a theme template that can be edited; a past edit may have added blanket disallows.
- Bot protection and JavaScript challenges that crawlers cannot pass.
- Rate limits that turn crawlers away during traffic peaks.
How to check your site in ten minutes
- Open
yourdomain.com/robots.txtand read every group. Look forDisallow: /under any AI token or under*. - Check your CDN or firewall dashboard for AI crawler or bot-fight settings.
- Search your server or CDN logs for OAI-SearchBot, PerplexityBot and Claude-SearchBot, and look at the status codes they received.
- In Google Search Console, inspect a key URL and confirm it is indexed and eligible for a snippet; Google says that is the only requirement for AI Overviews.
- Ask ChatGPT and Perplexity about your brand by name. If they never link your site, crawling is the first suspect.
What about llms.txt?
llms.txt is a proposed file that lists a site’s key pages for language models. It is cheap to add, but no major AI search engine has said it uses the file to decide what to cite, and Google’s AI features guidance requires nothing beyond normal Search eligibility. Treat it as optional; fix robots.txt and firewall access first.
You cannot be cited by an engine you have locked out. Check access before you rewrite a single page.
Not sure what your site is blocking?
Our free AI-search visibility check includes a crawler access review alongside a ten-prompt citation test.
Get the free checkFrequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot; allow it to appear in ChatGPT search answers.
Does Google-Extended affect AI Overviews?
No. Google states that Google-Extended does not affect inclusion in Google Search, and that Googlebot rules are the control for AI Overviews and AI Mode.
Which AI crawlers ignore robots.txt?
User-initiated fetchers may. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. The automated search and training crawlers respect robots.txt.
Should I block AI training crawlers?
It is a business choice. Blocking GPTBot, ClaudeBot and Google-Extended keeps your content out of future training without removing you from AI search answers, as long as the search crawlers stay allowed.
Related: How to get cited in AI Overviews, ChatGPT and Perplexity, GEO services, How to measure AI search visibility.