The short answer
WordPress serves a robots.txt for you, and plugins or a file in your site root can change it. Check what crawlers see first. Edit robots.txt in Yoast SEO under Tools, then File editor, or upload your own file. Yoast SEO can also generate an llms.txt from a site feature setting, including on its free plan.
- Start by checking your live robots.txt; plugins, hosts and firewalls can all change what crawlers see.
- Yoast SEO edits robots.txt under Tools, then File editor.
- Yoast SEO's free plugin can generate an llms.txt from its site features.
- Security plugins and host firewalls can block AI crawlers even when robots.txt allows them.
How do you see what AI crawlers see on WordPress?
Open yoursite.com/robots.txt in a browser. That is the file crawlers read, whatever produced it.
Run the free AI crawler checker on your domain. It shows which AI search, training and user crawlers each rule affects.
Also check Settings, then Reading. The option that discourages search engines from indexing the site should be unticked on a live site.
How do you edit robots.txt on WordPress?
Two common ways work. Use one, not both, so you know which file is live.
- 1
With Yoast SEO
Go to Yoast SEO, then Tools, then File editor. Create or edit robots.txt there and save. Your host must allow the plugin to write files.
- 2
With a file
Upload a plain-text file named robots.txt to your site root using your host's file manager or SFTP. A real file there replaces the one WordPress generates.
- 3
Allow AI search crawlers
Add groups for the AI search crawlers you want, like the sample below, and keep your existing rules.
- 4
Re-check
Reload yoursite.com/robots.txt, then run the crawler checker again.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
A named group replaces the general rules for that crawler. Copy any Disallow lines you still want, such as cart or account pages, into each group.
How do you add llms.txt on WordPress?
Yoast SEO includes an llms.txt feature, available in the free plugin. Turn it on in Yoast SEO's settings under site features. You can let it pick pages automatically or choose them yourself.
Without a plugin, write the file yourself and upload it to your site root as llms.txt. Then open yoursite.com/llms.txt to confirm it loads as plain text.
Yoast describes llms.txt as guidance for AI tools, not a ranking signal. It helps assistants find your most useful pages.
What else can block AI crawlers on WordPress?
robots.txt is only one layer. If the checker shows crawlers allowed but they never visit, look at these next.
- Security plugins with bot-blocking or rate-limit rules.
- Your host's firewall or bot protection.
- Cloudflare in front of your site, with AI bot policies set to block.
- Pages that only render content with JavaScript, which some crawlers cannot run.
Questions people ask
Does WordPress block AI crawlers by default?
WordPress itself does not add AI-specific blocks to the robots.txt it generates. Plugins, themes, hosts and Cloudflare can. Check your live robots.txt to be sure.
Do I need Yoast SEO for llms.txt?
No. llms.txt is a plain text file, so you can write and upload it yourself. Yoast SEO just generates and updates it for you.
Should I block GPTBot on WordPress?
GPTBot collects content that may train OpenAI's models. Blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot. Decide based on how you feel about training.