How to fix JavaScript-rendered content for AI crawlers

Why can JavaScript hide content from AI crawlers?

Some websites send a nearly empty HTML page and build the content in the browser. This is called client-side rendering. A person with a modern browser sees the full page.

A crawler is an automated program that fetches pages. Googlebot runs JavaScript, although rendering can happen later than the first fetch. Many AI crawlers are reported to read only the raw HTML.

Crawler behavior changes, so check each platform's documentation for the current position. The safe assumption is simple. If your key text is not in the raw HTML, some crawlers will not see it.

How to find and fix JavaScript-rendered content, step by step

Start with a few important pages, then roll the fix out across the site.

1. 1

   Pick your most important pages

   Choose your homepage, main service or product pages, pricing, About page and best-performing articles. These are the pages buyers and AI answers rely on most.

2. 2

   Fetch the raw HTML

   Use curl, or your browser's 'View page source', not the element inspector. Search the result for a sentence you know is on the page.

3. 3

   Compare with Google's rendered view

   In Search Console, use URL Inspection, then 'Test live URL' and 'View tested page'. This shows what Googlebot sees after running JavaScript.

4. 4

   List what is missing

   Note any headings, body text, prices, links or structured data that appear only after rendering. Missing navigation links matter too, because crawlers follow them.

5. 5

   Choose a rendering fix

   Move key pages to server-side rendering, static generation or prerendering. Most modern frameworks support at least one of these. Your developer can advise which fits your setup.

6. 6

   Move schema and meta tags server-side

   Structured data added by a tag manager may only exist after scripts run. Put JSON-LD, titles and meta descriptions in the server HTML instead.

7. 7

   Retest every changed template

   Repeat the raw HTML test on each page type after the change. Check again after every major site release.

What does a quick raw HTML test look like?

Run these commands in a terminal. Replace the address and the phrase with a real page and a sentence from it.

If grep prints nothing, the phrase is not in the raw HTML. Sending a crawler's name as the user agent shows what your server returns to that name. It does not prove what the real crawler receives.

# Raw HTML, as a crawler that does not run JavaScript sees it
curl -s https://www.example.com/services/ | grep -i "emergency plumbing"

# Same test, sending a crawler name as the user agent
curl -s -A "GPTBot" https://www.example.com/services/ | grep -i "emergency plumbing"

# Count the visible words in the raw HTML (rough check)
curl -s https://www.example.com/services/ | sed 's/<[^>]*>//g' | wc -w

Which rendering approach should you use?

The table compares the common options. The right choice depends on how often your content changes and what your framework supports.

How to check it worked

Run the raw HTML test again on every page you changed. Your key sentences, headings and links should now appear without JavaScript.

Use URL Inspection in Search Console to confirm Google still sees the full page. Then watch the Pages report for indexing changes over the next few weeks.

Finally, check that your structured data validates in the Rich Results Test. Test the live URL, not a pasted code snippet.

Common mistakes with JavaScript-rendered sites

These issues often survive a rendering fix, so check for them separately.

• Loading tab or accordion content only when someone clicks it.
• Using buttons with click handlers instead of real links with an href.
• Blocking JavaScript or CSS files in robots.txt, which breaks Google's rendering.
• Injecting structured data only through a tag manager.
• Loading articles through infinite scroll with no paginated URLs.
• Letting bot protection show crawlers a challenge page instead of content.

Questions people ask