The short answer
robots.txt tells crawlers which parts of your site they may visit. It is an established standard that major search engines and AI companies say they respect. llms.txt is a proposed Markdown file that points AI tools to your most useful pages. It controls nothing, and platform support is unconfirmed. You need robots.txt; llms.txt is optional, low-cost groundwork.
- robots.txt is about access: which crawlers may fetch which paths.
- llms.txt is about guidance: a short, curated map of your best content for AI tools.
- llms.txt cannot block or allow anything, and adoption by AI platforms is not confirmed.
- A wrong line in robots.txt can hide you from AI search; a missing llms.txt costs little.
What does robots.txt do?
robots.txt is a plain text file at the root of your site, such as example.com/robots.txt. It lists rules for crawlers by name, called user agents.
Each rule says which paths a crawler may or may not fetch. The format was formalized as an internet standard in 2022, after decades of informal use.
Compliance is voluntary, but major search engines and AI companies say they follow it. Here is a simple example that allows AI search crawlers and opts out of OpenAI model training.
User-agent: * Disallow: /admin/ # Allow ChatGPT search and Perplexity to index the site User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Opt out of OpenAI model training only User-agent: GPTBot Disallow: / Sitemap: https://www.example.com/sitemap.xml
What does llms.txt do?
llms.txt is a proposed standard published at llmstxt.org. It is a Markdown file served at /llms.txt.
It starts with an H1 of your name and a blockquote summary. H2 sections then list your key pages as links with short notes.
The goal is to give AI tools a quick, clean summary instead of a messy crawl. At the time of writing, no major AI platform has confirmed it reads the file when answering.
# Example Dental Clinic > A family dental clinic in Manchester offering check-ups, hygiene and implants. ## Services - [Dental implants](https://www.example.com/implants): Treatment, costs and recovery - [Emergency appointments](https://www.example.com/emergency): Same-day care and hours ## About - [Our dentists](https://www.example.com/team): Qualifications and experience
llms.txt vs robots.txt side by side
The two files sit next to each other at your site root, but they do very different jobs.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Control crawler access | Point AI tools to key content |
| Status | Established internet standard | Proposed standard, not widely confirmed |
| Format | Plain text rules by user agent | Markdown: H1, summary, H2 link lists |
| Can it block crawlers? | Yes, for crawlers that respect it | No |
| Who reads it | Search engines and AI crawlers | Some AI tools and agents; major platforms unconfirmed |
| Risk if wrong | High: can hide pages from search and AI | Low: outdated facts could mislead |
| Required? | Strongly recommended for every site | Optional |
Which one do you need?
Every site should have a correct robots.txt. It is the one file that directly decides whether AI search crawlers can reach you.
llms.txt is optional. It takes under an hour to write if your site is small. Treat it as cheap groundwork, not a lever that changes AI answers.
Each AI company now runs several crawlers with different jobs. OpenAI, Anthropic and Perplexity each separate search or user fetches from training. That lets you make a separate choice for each purpose.
- Check robots.txt first: are OAI-SearchBot, PerplexityBot and Googlebot allowed?
- Decide separately whether to allow training crawlers like GPTBot and ClaudeBot.
- Add llms.txt if you can keep it accurate as your site changes.
You can opt out of AI training and still allow AI search: give each crawler its own rule.
Common mistakes with robots.txt and llms.txt
Most problems come from copying rules without checking what they do.
- Blocking all AI bots with one rule, which can also block AI search, not just training
- Thinking that blocking GPTBot removes you from ChatGPT search; OAI-SearchBot controls that
- Using llms.txt to try to block crawlers; it has no blocking power
- Listing pages in llms.txt that redirect, are missing or contradict your live site
Before you block any AI bot, check whether it collects training data or powers a search product.
The verdict: robots.txt first, llms.txt as a bonus
robots.txt is essential plumbing. Get it right and review it whenever you change hosting or security settings.
llms.txt is a reasonable bet on a standard that may or may not catch on. Write it once, keep it short and keep it true.
WebValet's free check looks at both. It reports how your robots.txt treats AI bots and whether you have an llms.txt file.
Questions people ask
Can llms.txt stop AI companies from using my content?
No. llms.txt is a guide, not an access rule. To limit crawling, use robots.txt rules for each named crawler.
Does ChatGPT read llms.txt?
OpenAI has not confirmed that ChatGPT uses llms.txt when answering, at the time of writing. Check OpenAI's documentation for updates.
Will a firewall or CDN override robots.txt?
It can. Some security settings block AI bots before they ever read robots.txt. Check your CDN or firewall bot settings as well as the file itself.
Should robots.txt link to llms.txt?
There is no standard way to do that. Robots.txt can list your sitemap with a Sitemap line. llms.txt simply lives at /llms.txt.