The short answer
An AI readiness audit checks whether AI crawlers can reach, read and understand your site. Review robots.txt and firewall rules for AI bots, and confirm key text is in the raw HTML. Validate your structured data and check page speed. Then make sure your homepage states plainly who you are, what you offer and where.
- Readiness means crawlers can reach, read and understand your site.
- Firewall and CDN bot settings can block AI crawlers even when robots.txt allows them.
- Training crawlers and search crawlers are separate, so you can treat them differently.
- Being ready does not guarantee AI answers will recommend you.
What does AI readiness mean?
AI readiness is how easily AI systems can access and understand your website. It has four layers: access, readability, understanding and trust.
Access means crawlers are allowed in. Readability means your text is in the HTML. Understanding means your pages clearly say what you do. Trust comes from proof on your site and mentions elsewhere.
This audit covers the first three layers. Trust needs work beyond your own site, such as reviews, directories and press.
How to audit your site for AI readiness, step by step
You need a browser, Search Console access and, ideally, a developer for fixes.
- 1
Read your robots.txt
Open yourdomain.com/robots.txt. Look for Disallow rules that name AI crawlers, or a blanket rule that blocks all bots.
- 2
Check firewall and CDN settings
Some hosting and CDN providers offer one-click AI bot blocking. Check whether it is on, and whether that matches your intent.
- 3
Check the raw HTML
Fetch key pages with curl or view the page source. Confirm headings, body text and links are present without JavaScript.
- 4
Check sitemaps and indexing
Make sure your XML sitemap lists your key pages. Review indexing in Search Console and Bing Webmaster Tools, since Copilot draws on Bing.
- 5
Validate structured data
Test key pages in Google's Rich Results Test or the Schema Markup Validator. Fix errors and confirm the facts match the page.
- 6
Review llms.txt
llms.txt is a proposed standard for a Markdown summary of your site. AI platforms have not confirmed they use it, so treat it as optional.
- 7
Test speed
Run key pages through PageSpeed Insights. Slow servers and heavy pages can cause crawlers to fetch less.
- 8
Read your homepage as a stranger
Within the first screen, can a newcomer tell who you are, what you offer and where? If not, rewrite the opening.
Which AI crawlers should the audit cover?
Each company runs separate crawlers for different jobs. Names change, so confirm them in each company's official documentation.
| Name | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects training data |
| OAI-SearchBot | OpenAI | Builds the ChatGPT search index |
| ChatGPT-User | OpenAI | Fetches pages when a user asks |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
| Perplexity-User | Perplexity | Fetches pages when a user asks |
| ClaudeBot | Anthropic | Collects training data |
| Claude-SearchBot | Anthropic | Indexes pages for search |
| Claude-User | Anthropic | Fetches pages when a user asks |
| Google-Extended | A robots.txt token for Gemini training and grounding, not a crawler | |
| Googlebot | Google Search, including AI Overviews | |
| Bingbot | Microsoft | Bing index, which Copilot draws on |
What does an AI-ready robots.txt look like?
This fictional example allows AI search and user-triggered fetches. It opts out of training, which is a business choice, not a requirement.
Blocking GPTBot does not remove you from ChatGPT search; OAI-SearchBot controls that. Blocking Google-Extended does not affect Google Search or AI Overviews.
# Allow AI search and user-triggered fetches User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User Allow: / # Example only: opt out of training (and Gemini grounding for Google-Extended) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: / Sitemap: https://www.example.com/sitemap.xml
How to check it worked
Reload robots.txt in a private window to confirm the live version changed. Some CDNs cache it.
Check your server or CDN logs for visits from AI crawlers over the following weeks. Several companies publish IP ranges, so you can confirm visits are genuine.
WebValet's free check runs these core tests in about 20 seconds, with no signup. It reviews robots.txt rules for AI bots, structured data, llms.txt, homepage clarity and speed.
Common mistakes in AI readiness audits
These are easy to miss on a first pass.
- Checking robots.txt but not the firewall or CDN bot settings.
- Blocking GPTBot and assuming that removes you from ChatGPT search.
- Treating llms.txt as a fix for crawler access problems.
- Testing only the homepage and not service or product pages.
- Confusing readiness with being recommended.
Questions people ask
Should I block AI training crawlers?
It is a business decision. Blocking training crawlers keeps your content out of future training sets. Search and user-triggered crawlers are separate, so you can still allow those.
Do I need an llms.txt file?
It is optional. llms.txt is a proposed standard, and AI platforms have not confirmed they use it. It is cheap to add, but fix access and content first.
How often should I repeat the audit?
Run a full audit after any redesign, migration or hosting change. Otherwise, a quick monthly check of robots.txt and key pages is enough for most sites.
Does the free WebValet check ask AI platforms about my brand?
Not today. The free check tests site readiness for Google and AI crawlers. Live AI answer checks are rolling out, and Pro tracks buyer prompts weekly as that rollout continues.