The short answer
robots.txt is a plain text file at your site's root that tells crawlers which pages they may fetch. Search engines and most AI companies follow it voluntarily. It controls crawling, not indexing or security. For AI search, it decides which AI bots can read your pages and later cite them.
- robots.txt lives at yourdomain.com/robots.txt and sets crawl rules per bot.
- A bot with its own group ignores the general User-agent: * group.
- It is not security: it asks politely, and it does not remove pages from search.
- One wrong line can block AI search bots, or every bot, by accident.
How does robots.txt work?
Before crawling a site, a polite bot reads its robots.txt file. The file contains groups of rules, each starting with a User-agent line.
The protocol is old, but it was formally written up as an internet standard, RFC 9309, in 2022.
Rules are matched against the path of each URL. A Disallow: /cart/ line covers every page that starts with /cart/. Lines starting with # are comments that bots ignore.
Bots that cannot find the file at all treat the whole site as open. That is usually fine, but it gives you no control.
# Default rules for all bots User-agent: * Disallow: /admin/ Disallow: /cart/ # A specific rule for one bot User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
What do the robots.txt rules mean?
A handful of directives cover almost every case.
| Directive | What it does |
|---|---|
| User-agent | Names the bot a group of rules applies to; * means all bots |
| Disallow | A path the bot should not fetch |
| Allow | A path the bot may fetch, even inside a disallowed folder |
| Sitemap | Points crawlers to your XML sitemap |
| Crawl-delay | Unofficial; asks for a pause between requests; Google ignores it |
When Allow and Disallow both match a URL, the longer, more specific rule wins.
Why does robots.txt matter for AI search?
AI companies name their crawlers so you can treat them differently. You might allow OAI-SearchBot and PerplexityBot but block training bots.
There is one trap. A bot that has its own group follows only that group, and ignores the * group entirely.
So if you add a group for GPTBot, repeat any general blocks you still want it to respect. Otherwise private folders may become crawlable for that bot.
What robots.txt cannot do
Many problems come from expecting too much of the file.
- It is not security. Anyone can read it, and rude bots can ignore it.
- It does not remove pages from search. A blocked URL can still be indexed from links elsewhere.
- It cannot apply noindex. Search engines must crawl a page to see a noindex tag.
- It is not retroactive. Blocking a bot today does not delete what it collected before.
- It does not override firewalls. A CDN rule can block a bot that robots.txt allows.
How to check your robots.txt
Open yourdomain.com/robots.txt in a browser and read it line by line. Each subdomain needs its own file.
Google Search Console has a robots.txt report showing the version Google fetched and any errors.
Also check your CDN or firewall. Many now include bot controls that act before robots.txt is even read.
WebValet's free check reads your rules for the main AI bots and flags any that are blocked.
Common robots.txt mistakes
These show up on live sites more often than you would think.
- Leaving Disallow: / in place after launching from a staging site.
- Blocking CSS and JavaScript files that search engines need to render pages.
- Misspelling bot names, so the rule never applies.
- Forgetting that paths are case-sensitive.
- Copying a template that blocks AI search bots you actually want.
A sample robots.txt for AI search visibility
Here is a fictional starting point for a small business site. It keeps private areas closed to every bot. It welcomes AI search bots and opts out of two training bots.
Notice that the AI search group repeats the private folders. That is because those bots follow only their own group.
User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Claude-SearchBot Disallow: /admin/ Disallow: /cart/ User-agent: GPTBot User-agent: ClaudeBot Disallow: / Sitemap: https://example.com/sitemap.xml
Adjust the training rules to your own choice; allowing them is equally valid.
Questions people ask
Where does robots.txt need to live?
At the root of each host, such as example.com/robots.txt. A file in a subfolder is ignored. Each subdomain needs its own file.
Does every website need a robots.txt file?
No. Without one, crawlers assume everything is allowed. A file is still useful for blocking private areas and pointing to your sitemap.
How quickly do robots.txt changes take effect?
Google generally caches robots.txt for up to a day. Other crawlers have their own timing, so allow a few days for everything to catch up.
Is robots.txt legally binding?
It is a widely respected convention, not a technical barrier. Legal questions vary by country, so ask a lawyer if you need a firm answer.
Should I block AI bots in robots.txt?
Block AI search bots only if you do not want to be cited by those tools. Training bots are a separate choice about how your content is used. Decide bot by bot rather than with one blanket rule.