How to make your site readable to AI agents

Four changes, in descending order of payoff: put the content in the server-rendered HTML, serve a markdown version at the same URL under Accept: text/markdown, advertise that version with an RFC 8288 Link header, and keep one H1 with headings that do not skip levels. The first one is worth more than the other three combined, and a client-only JavaScript app fails it completely.

None of this helps you in Google. Google's generative AI guidance, updated 10 July 2026, says Search does not use extra machine-readable files and processes JavaScript fine. This post is about the crawlers that are not Google.

Key Takeaways

• The last public measurement at scale, Vercel's December 2024 crawler analysis, found no major AI crawler executing JavaScript. Nothing comparable has been published since.
• Content negotiation on Accept: text/markdown gives agents a clean copy at the same URL, so citations still point at a page a human can read.
• Link headers are set in server or CDN config, work on non-HTML responses, and need no template changes.
• Serve both formats. Do not replace your HTML page with markdown.

Server-render the content, or nothing else matters

An agent that cannot see your text will not be helped by a header pointing at more of it.

The last measurement of this at any scale is Vercel's crawler analysis, published 17 December 2024, drawn from traffic on its own network. Its finding: "none of the major AI crawlers currently render JavaScript. This includes: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot)... Perplexity (PerplexityBot)." ChatGPT's crawlers fetched JavaScript files in 11.50% of requests and Claude's in 23.84%, and executed none of them.

That study is 20 months old at the time of writing, which matters. What has not changed is that no crawler operator documents JavaScript rendering support in its own crawler documentation. OpenAI's and Perplexity's crawler pages describe robots.txt behaviour and say nothing about rendering. Absence of a claim is not proof, and I am not going to pretend otherwise. But the only public measurement and the total absence of vendor claims point the same way, so put the text in the server response and stop thinking about it.

The test costs nothing:

$ curl -s https://yoursite.com/your-page | grep -c "a distinctive sentence from your page"

If that returns 0, your content is not in the HTML. A client-only React or Vue app with no server rendering will return an empty shell to every one of those crawlers. Our AI content extractor does the same check in a browser tab and shows you the text that survives without JavaScript.

Serve markdown at the same URL

Content negotiation is the oldest idea in HTTP and it fits this problem exactly. The client says what it can read, the server answers in kind, and the URL stays the same, so an agent that cites your page cites a URL a human can open.

text/markdown has been a registered media type since RFC 7763 in March 2016, so there is nothing to invent here.

Cloudflare shipped this as a product feature. Their own documentation answers it:

$ curl -sI -H "Accept: text/markdown" \
    https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
HTTP/2 200
content-type: text/markdown; charset=utf-8
vary: accept-encoding, accept

The same URL with a browser's Accept string returns content-type: text/html. Note the vary: accept in that response. Without it, a shared cache will hand the markdown to the next human who asks for the page.

We do it too, and you can run this yourself:

$ curl -s -H "Accept: text/markdown" https://seobuilder.tech/pricing | head -12
[SEOBuilder](/) [Features](/#features) [How It Works](/#how-it-works) [Free tools](/tools) [Pricing](/pricing)

# Real tools. Real prices.

Compete with the big tools, without the big price tag, or the per-seat tax,
or the hidden AI upcharge. No hidden fees. Cancel anytime.

Every plan includes the full feature set, not a cut-down tier AI insights
included, not a $195 addon Monthly AnnualSave 20%

### Starter

There are two ways to produce that markdown, and the choice has consequences.

Derive it from your own HTML. That is what happens above. Our proxy rewrites the request to a route that fetches the page's own HTML and converts it. The output is imperfect (look at the run-on line about the $195 addon, which is two adjacent spans in the markup with no whitespace between them) but it cannot drift. A hand-maintained second copy of your pricing page will eventually disagree with the real one, and an agent has no way to tell which copy is current.

Serve the source file. Where the page was rendered from markdown in the first place, hand over the original. Blog posts on this site do that: the file in the repository is what an agent gets, headings, code fences and all, rather than a reconstruction of the rendered HTML.

Derived output is honest but lossy. Source output is exact but only available when a source exists. Use source where you have it and derivation everywhere else, and never maintain a second copy by hand.

One limit, stated plainly: I could not find a published figure, from Cloudflare or anyone else, for how many agent requests actually send Accept: text/markdown. The CDNs are in a position to count it and none of them has said. So the case for doing this is that it is cheap and cannot hurt, not that a measured share of traffic is asking for it.

Advertise it with a Link header

RFC 8288, Web Linking, October 2017, Standards Track, is the spec for the Link response header. It carries typed relationships to other resources without touching the page body, which means three useful properties: it works on non-HTML responses, it can be set in CDN or web server configuration without editing a template, and a client can read it from the response headers alone before parsing anything.

The llms.txt v2 spec recommends exactly two relations for this:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown",
      </docs/llms.txt>; rel="describedby"

rel="alternate" with type="text/markdown" says "the same resource, in this format". rel="describedby" points at the file that describes this page. Neither is new: both are entries in the IANA link relation registry, alternate from the HTML spec and describedby from POWDER.

Ours currently advertises the machine-readable descriptors rather than a markdown twin:

$ curl -sI https://seobuilder.tech/ | grep -i ^link
link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json",
 </auth.md>; rel="service-doc"; type="text/markdown",
 </.well-known/mcp/server-card.json>; rel="service-desc"; type="application/json",
 </sitemap.xml>; rel="sitemap"; type="application/xml"

Two things in that header are worth copying and one is not. api-catalog, service-doc and service-desc are registered relations, from RFC 9727 and RFC 8631. sitemap is not in the registry at all, which I only noticed while checking the other three for this post. It is harmless and widely used, but it is a convention we adopted without checking, which is the same mistake this cluster keeps warning about.

The real bug is the scope. That header is configured for / only. Run the same command against https://seobuilder.tech/pricing and you get nothing back, even though /pricing is one of the pages that answers Accept: text/markdown. An agent that arrives from a search result on a deep page never sees the pointer, because we only put the sign on the front door. Set the header on every route that has an alternate, not on the homepage.

Check what your own pages return with the HTTP header checker.

Headings that an answer can point at

The last change is the least technical. One H1, H2s under it, H3s under those, no skipped levels. That advice is 20 years old, and it earns its place here because an extractive answer wants to quote a passage, and a passage needs an address.

Give every heading a stable id so a citation can land on the section rather than the page. This post's HTML does that, which is why the anchors in the table of contents work. If two sections share a title, suffix the second id rather than letting the first one shadow it, or half your page becomes unreachable by fragment.

Two things that quietly break extraction, both common:

• Text baked into images or SVG with no accessible equivalent. It is invisible to a text pipeline.
• A heading used for visual size rather than structure. An H2 that says "Ready to get started?" above a signup form tells a parser that a section about getting started begins there.

What to change, in what order

1. Fetch your key pages with curl and confirm your text is in the response body. Fix server rendering before anything else.
2. Add Accept: text/markdown handling on the pages that are identical for every visitor. Never on anything behind a session.
3. Send Vary: Accept with it, or a shared cache will serve the wrong format to somebody.
4. Set the Link header on every route that has an alternate, not just the homepage.
5. Read what llms.txt actually is and who reads it before you spend an afternoon on one, and check the crawler you care about is the one you think it is in Google-Extended vs Googlebot.

Then measure. SEOBuilder tracks Google AI Overviews and nothing else, so it will tell you whether these pages get cited there and will tell you nothing about ChatGPT or Perplexity. Start with how to measure AI visibility, or score the whole set of signals with the free agent readiness check.