The short answer
Open the robots.txt file at your site root. Add a group naming the AI search crawlers you want, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot. Give it an Allow rule. Remove old rules that block them. Decide separately whether training crawlers like GPTBot may visit. Then test the live file and check your firewall is not blocking them anyway.
- AI companies run separate crawlers for search, training and user-requested fetches.
- Blocking GPTBot does not remove you from ChatGPT search; OAI-SearchBot controls that.
- A crawler named in its own group ignores your User-agent: * rules, so repeat private paths.
- Firewalls and CDN bot settings can block AI crawlers even when robots.txt allows them.
Which AI crawlers should you know about?
A crawler is a program that fetches web pages automatically. Each one identifies itself with a name called a user agent.
Most AI companies run several crawlers with different jobs. Some build a search index. Some collect training data. Some fetch a page because a user asked about it.
These names change from time to time. Check each company's official crawler documentation before you publish.
| Company | User agent | What it does |
|---|---|---|
| OpenAI | OAI-SearchBot | Builds the index for ChatGPT search |
| OpenAI | GPTBot | Collects data for model training |
| OpenAI | ChatGPT-User | Fetches a page when a user asks |
| Perplexity | PerplexityBot | Indexes pages for Perplexity answers |
| Perplexity | Perplexity-User | Fetches a page when a user asks |
| Anthropic | Claude-SearchBot | Indexes pages for Claude search |
| Anthropic | ClaudeBot | Collects data for model training |
| Anthropic | Claude-User | Fetches a page when a user asks |
| Google-Extended | A control token for Gemini training and grounding, not a separate crawler | |
| Microsoft | Bingbot | Builds the Bing index, which Copilot answers draw on |
Google-Extended does not affect Google Search or AI Overviews. Those use Googlebot and normal Search controls.
How to allow AI crawlers, step by step
This takes about 15 minutes if you can edit files on your site or in your CMS.
- 1
Open your current file
Visit yoursite.com/robots.txt. Copy the contents somewhere safe before you change anything.
- 2
Look for blocks
Search for Disallow: / under User-agent: * or under any AI crawler name. Old staging rules often hide here.
- 3
Decide your training policy
Search crawlers help you appear in answers. Training crawlers are a separate choice about how your content is used.
- 4
Add a group for AI search crawlers
List several User-agent lines in one group, then Allow: /. One group can name many crawlers.
- 5
Repeat your private paths
A named crawler skips the * group. Copy Disallow lines for admin, cart or account areas into each named group.
- 6
Keep your sitemap line
End the file with Sitemap: and the full URL of your XML sitemap.
- 7
Check your firewall and CDN
Some hosts and CDNs offer one-click AI bot blocking. Make sure those settings match your robots.txt.
A robots.txt example that allows AI search crawlers
This example allows AI search and user fetchers, and also allows training crawlers. To opt out of training, change that group's Allow to Disallow.
Swap the example paths for the private areas on your own site.
# AI search crawlers and user-triggered fetchers User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /cart/ # Training crawlers: change Allow to Disallow to opt out User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /cart/ # Everyone else, including Googlebot and Bingbot User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
How to check it worked
First, reload yoursite.com/robots.txt and confirm the new rules are live. Caches can serve the old file for a while.
Next, check your server or CDN logs over the following weeks. Look for requests from the crawler names above, returning status 200.
User agents can be faked. OpenAI and others publish IP ranges, so verify important visitors against those lists.
Some platforms say user-triggered fetchers may not follow robots.txt, because a person asked for the page. Read each platform's documentation for current rules.
The free WebValet check reads your robots.txt and flags rules that block the main AI crawlers.
# Quick test: does your server or firewall reject this user agent? curl -I -A "OAI-SearchBot" https://example.com/ # A 403 or a challenge page suggests a firewall rule is blocking it.
Common robots.txt mistakes with AI crawlers
These errors are easy to make and easy to miss.
- Leaving a staging rule of Disallow: / on the live site.
- Blocking GPTBot and assuming you have left ChatGPT search, or the reverse.
- Forgetting that named groups ignore the * group, which exposes private paths.
- Allowing crawlers in robots.txt while a firewall still blocks them.
- Using robots.txt to hide sensitive pages. The file is public and is not security.
- Blocking a page you want removed from Google. Use a noindex tag instead.
Questions people ask
If I allow AI crawlers, will AI answers cite me?
Not automatically. Access is only the first step. Answers also depend on your content, your reputation and the sources that mention you.
Should I block AI training crawlers?
It is a business choice. Blocking GPTBot or ClaudeBot limits training use but does not, by itself, remove you from their search features. Many brands allow search crawlers and decide on training separately.
Does Google-Extended control AI Overviews?
No. Google-Extended only governs use of your content for Gemini training and grounding. AI Overviews use Googlebot and normal Search controls, such as nosnippet.
How do I make sure Microsoft Copilot can see my site?
Copilot answers draw on the Bing index. Allow Bingbot, submit your sitemap in Bing Webmaster Tools, and fix any crawl errors it reports.