The short answer
Webflow keeps both files in Site settings under SEO. Write robots.txt rules in the Indexing section, where traffic control toggles can also add rules for known bot groups. Upload llms.txt in the LLMs.txt section as a UTF-8 file under 100 KB. Publish, then check both on your custom domain.
- robots.txt rules live in Site settings, then SEO, then Indexing.
- Traffic control toggles in the same section can add bot rules for you, so check what they generate.
- Upload llms.txt in Site settings, then SEO, then LLMs.txt.
- Both files go live on your custom domain after you publish, not on the staging domain.
Which AI crawlers should you allow?
AI companies run separate crawlers for search and for training. Search crawlers decide whether AI answers can find, cite and link you.
Training crawlers collect content that may train future models. Blocking them does not remove you from AI search.
| Keep allowed to be cited | Training crawlers, your choice |
|---|---|
| OAI-SearchBot (ChatGPT search) | GPTBot (OpenAI) |
| Claude-SearchBot (Claude search) | ClaudeBot (Anthropic) |
| PerplexityBot (Perplexity) | CCBot (Common Crawl) |
| Googlebot and Bingbot (Google, Copilot) | Google-Extended, Applebot-Extended (control tokens) |
How do you set robots.txt rules in Webflow?
Webflow's help center places robots.txt in the site's SEO settings.
- 1
Open the settings
Go to Site settings, then SEO, then the Indexing section.
- 2
Review traffic control
Check any traffic control toggles. Webflow says they generate rules for known bot groups, which may include AI crawlers.
- 3
Add your rules
Add groups for the AI search crawlers you want to allow, keeping any rules you need.
- 4
Save and publish
Save the changes and publish the site so the new file goes live.
- 5
Check it
Open yourdomain.com/robots.txt and run the AI crawler checker.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
A named group replaces the general rules for that crawler. Copy any Disallow lines you still want, such as cart or account pages, into each group.
What is the Content-Signal setting in Webflow?
Webflow's help center says it can serve a Content-Signal header that states your AI usage preferences. It is configured in the same Indexing section.
Content signals express how you want content used, such as for search or training. Like robots.txt, they are a stated preference, not an enforced block.
How do you add llms.txt in Webflow?
Webflow describes llms.txt as closer to a sitemap than to robots.txt. It guides AI tools to your best pages and blocks nothing.
- 1
Write the file
Write llms.txt as plain Markdown text, saved as UTF-8. Keep it under 100 KB, Webflow's limit.
- 2
Upload it
Go to Site settings, then SEO, then LLMs.txt, and upload the file.
- 3
Publish and check
Publish to your custom domain, then open yourdomain.com/llms.txt. Webflow serves it on the custom domain, not the staging domain.
Questions people ask
Does Webflow block AI crawlers by default?
Check your Indexing settings and live robots.txt. Traffic control toggles can add bot rules, so a setting someone turned on earlier may block AI crawlers.
Can I generate llms.txt inside Webflow?
Webflow's documented feature is an upload. Write the file yourself or with a generator, then upload it in the LLMs.txt section.
Why can't I see llms.txt on my webflow.io address?
Webflow serves the uploaded file at the root of your custom domain, not on the staging domain.