The short answer
AI crawlers are automated programs that AI companies use to fetch web pages. Some collect training data, and some build search indexes for AI answers. Others fetch a page when a user asks. Each has its own user-agent name, so you can allow or block them separately in your robots.txt file.
- AI crawlers do three jobs: training, search indexing and user-triggered fetches.
- Blocking a training bot does not usually remove you from that company's AI search.
- Blocking an AI search bot makes it hard for that tool to cite your pages.
- Firewalls and CDN bot settings can block AI crawlers even when robots.txt allows them.
What jobs do AI crawlers do?
AI companies now split their bots by purpose. That lets site owners make finer choices.
A user agent is the name a bot gives when it requests a page. It is how robots.txt rules target one bot.
| Job | What it does | Examples |
|---|---|---|
| Training | Collects content that may train future models | GPTBot, ClaudeBot, CCBot |
| Search indexing | Builds the index AI search answers draw on | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| User-triggered | Fetches a page because a user asked | ChatGPT-User, Perplexity-User, Claude-User |
Why do AI crawlers matter for visibility?
If an AI search bot cannot read your pages, that tool struggles to cite them. Your competitors' pages fill the gap.
Training bots are a different decision. Some businesses opt out for content or licensing reasons. Others allow them so models learn about their brand.
Either choice is valid. The key is making it on purpose, bot by bot.
User-triggered fetchers deserve thought too. When a buyer pastes your link into ChatGPT or asks Perplexity about your page, these agents do the reading. Blocking them can make your page look unavailable at a key moment.
Which AI crawlers and tokens should you know?
These names are current at the time of writing. Companies add and rename bots, so check each official page before editing rules.
| Name | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ChatGPT-User | OpenAI | Fetches pages when a user asks |
| PerplexityBot | Perplexity | Search index for answers |
| Perplexity-User | Perplexity | Fetches pages when a user asks |
| ClaudeBot | Anthropic | Training data |
| Claude-SearchBot | Anthropic | Search for Claude |
| Claude-User | Anthropic | Fetches pages when a user asks |
| Google-Extended | Control token for Gemini training and grounding, not a crawler | |
| Bingbot | Microsoft | Bing index, which Microsoft Copilot draws on |
| CCBot | Common Crawl | Open web archive that many AI developers use |
How to control AI crawlers in robots.txt
This sample allows AI search and user fetches, but opts out of training. Adjust it to your own choices.
# Allow AI search and user-triggered fetches User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Claude-SearchBot User-agent: Claude-User Allow: / # Opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: /
robots.txt is voluntary: reputable bots follow it, but it is not a lock.
How to check what AI crawlers can access
WebValet's free check reads your robots.txt rules for the main AI bots and flags any that are blocked.
- 1
Read your robots.txt
Open yourdomain.com/robots.txt and look for AI bot names and broad blocks.
- 2
Check your CDN and firewall
Some hosting and CDN services offer one-click AI bot blocking. Check what is switched on.
- 3
Look at server logs
Search for AI user-agent names to see which bots actually visit.
- 4
Verify real bots
User agents can be faked. Several AI companies publish IP ranges you can check against.
How AI crawlers differ from search engine crawlers
Googlebot and Bingbot have crawled the web for decades. They follow well-documented rules and render most pages fully.
AI crawlers are newer and less uniform. Many appear to fetch only the raw HTML, without running JavaScript. Some visit in bursts rather than at a steady pace.
The overlap still matters. Google's AI features use Googlebot, and Microsoft Copilot draws on Bing. Good search crawlability is part of AI visibility too.
Common AI crawler mistakes
These are the issues that most often hide a site from AI answers.
- A security plugin or CDN setting that blocks all AI bots by default.
- A copied robots.txt that blocks search bots along with training bots.
- Key facts that only appear after JavaScript runs.
- Rules that use outdated bot names and miss the current ones.
- Never checking server logs to see which bots actually arrive.
Questions people ask
Should I block AI crawlers?
It depends on your goals. Most businesses that want AI visibility allow AI search bots. Blocking training bots is a separate choice about how your content is used.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, which you control separately.
Do AI crawlers run JavaScript?
Many appear to read only the raw HTML a server sends. Content that loads only through JavaScript may be missed. Server-side rendering is the safer choice for key pages.
Can bots pretend to be AI crawlers?
Yes. Anyone can copy a user-agent name. Check suspicious traffic against the IP ranges that AI companies publish.
Do AI crawlers slow down my website?
Usually not noticeably, but heavy crawling can strain small servers. Caching and a CDN help most. Some bots also honor a Crawl-delay rule, so check each company's documentation.