llms.txt: what the spec says and who actually reads it

llms.txt is a proposal, not a standard, and as of 15 August 2026 no major AI search product documents that it reads yours. Google says outright that Search ignores the file. OpenAI, Anthropic and Google all publish an llms.txt for their own developer documentation, which is a different act entirely. Publishing one is not consuming one.

That does not make the file worthless. It makes it a bet with a known cost and an unproven payoff, which is a fine thing to place as long as you know that is what you are doing.

Key Takeaways

• The spec is at llmstxt.org, authored by Jeremy Howard, published 3 September 2024 and updated to v2 on 10 August 2026.
• Google's generative AI guidance states that Google Search does not use llms.txt. No other major vendor documents reading it either.
• Chrome's Lighthouse now audits for the file, but marks a missing file "not applicable" because providing it is optional.
• The more useful half of v2 is not the index file. It is markdown versions of your pages, discoverable through Link headers.
• Our own site did not have one, and /llms.txt was answering with a login redirect. Details below.

What the v2 spec actually requires

The file is markdown, and the structure is fixed enough to parse with a regex. In order: an optional byte order mark, an H1 with the site name (the only required section), a blockquote holding a short summary, then any number of non-heading markdown sections, then any number of H2 sections each containing a list of links.

# SEOBuilder

> Rank tracking, site audits and AI visibility scans for one person, not a team.

Prices and limits on this site are generated from one source, so the
numbers here match the numbers a customer is charged.

## Docs

- [How an agent authenticates](https://seobuilder.tech/auth.md): API keys, scopes, the MCP endpoint

## Optional

- [Terms](https://seobuilder.tech/terms)

Each list item is a markdown link followed by an optional colon and a note. The section named "Optional" is a convention, not a keyword: it marks links an agent can skip when context is tight.

v2 changed two things worth knowing. The file no longer has to sit at the root. /docs/llms.txt covers everything under /docs/, and where several apply, the most specific one wins. That is a deliberate rejection of the /.well-known/ approach, and the spec says why: well-known URIs exist only at the origin root, so anyone who controls a path but not a host, a GitHub Pages project for instance, could never publish one.

The second change is the one the tooling has mostly ignored. Alongside the index file, v2 proposes serving a clean markdown version of each page at the same URL with .md appended or substituted, and pointing at both with standard link relations:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

That second proposal is worth more than the index file, for reasons this post returns to at the end.

Which AI products have confirmed they read it

None of them. Here is the state of the evidence, checked on 15 August 2026.

Google is the only one that has answered the question directly, and the answer is no. From Google's guide to optimizing for generative AI features, last updated 10 July 2026: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

OpenAI's crawler documentation names four agents: OAI-SearchBot for ChatGPT search, OAI-AdsBot for ad landing pages, GPTBot for training, and ChatGPT-User for user-initiated fetches. It describes what each one does with robots.txt. It mentions llms.txt exactly once, in a line at the top of the page pointing at OpenAI's own documentation index. Perplexity's crawler page does the same thing with the same file, for its own docs.

So four companies publish one and none of them says it reads yours. That gap is what the spec was designed for. The file is a menu you hand to an agent already looking at your site, and the spec never claimed it was a signal you broadcast at a crawler.

The strongest institutional nod so far is from Chrome. Lighthouse gained an Agentic Browsing category (experimental, Chrome 150 and later) and one of its checks is llms.txt. Read what the check does before treating it as an endorsement. It flags a server error when fetching the file. If the file is missing and the server returns 404, the audit is marked Not Applicable, because, in Google's words, "providing the file is optional at the moment." A check that treats absence as not applicable is testing whether your server is broken, not whether your site is better.

The reader that does exist

The spec is explicit about its own model: "Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links."

That is a user-initiated fetch. You paste a library's URL into a coding agent, the agent looks for a map, and the map saves it from crawling forty HTML pages to find the API reference. This is why documentation platforms adopted the file first. The spec's own integrations list names Mintlify, GitBook, Wix, Yoast SEO and AIOSEO as generating one automatically, and llms.txt files are, in the author's words, "used most heavily for software documentation."

That path is real, and if you ship developer docs you should have one. If you run a 12-page marketing site, the path to a reader is much longer and nobody has published a measurement of it.

We did not have one, and the URL was worse than empty

Here is our own miss, checked before publishing this post.

$ curl -sI https://seobuilder.tech/llms.txt
HTTP/2 307
location: /login?redirect_url=%2Fllms.txt
x-clerk-auth-reason: session-token-and-uat-missing

There is no llms.txt in this repository at all. Because the file does not exist, the request fell through to our auth middleware, which protects every route not on an explicit allowlist, and Clerk sent a signed-out visitor to the sign-in page. An agent following that redirect gets a login form where it expected a site map.

The reason this survived is that it looks fine in a browser. We are signed in. A signed-in request for /llms.txt does not redirect, it 404s from the app, which at least reads as "no file here". Signed out, it looks like a page that exists and requires an account.

There is a second version of this failure that is more common and harder to spot: a framework catch-all route answering /llms.txt with a 200 and your HTML shell. Status 200, content-type text/html, body full of <div id="root">. Every validator that only checks for a 200 will pass it.

Check yours with curl while signed out, and read the status line and the content type, not the rendered page. If you want it done in a browser tab, our HTTP header checker shows the raw response, and the llms.txt validator parses an existing file against the spec and fetches every link in it to find the dead ones.

So is it worth publishing?

The expensive part of an llms.txt is keeping it true, not the twenty minutes of writing. A stale one is worse than none: a curated map with your name on it, handed to a machine that has no way to know a third of it now 404s.

Publish one if your site has documentation, if you can generate it from a source that already exists, and if something in your build breaks when a link in it dies. Skip it if the honest answer is that you will write it once, feel productive, and never open the file again.

And if you only have one hour, spend it on the other half of v2 instead. Serving a clean markdown version of each page is useful to an agent no matter how it arrived, whether it read your index file, followed a search result, or was handed the URL by a person. It needs no adoption from anyone. We do this for every post on this blog: the same URL under Accept: text/markdown returns the markdown source file this page was rendered from, not a reconstruction of the HTML.

To be clear about our own position: SEOBuilder asks seven answer engines, ChatGPT, Perplexity and Claude among them, so we can tell you whether those answers cited you. What we still cannot tell you is whether an llms.txt is the reason, and neither can anyone else, because none of those products reports what it read. Citation is measurable and cause is not. Anyone selling you an llms.txt on a promise of AI citations is selling you a file, not a result.

What to do this week

1. Run curl -sI https://yoursite.com/llms.txt signed out. If it is 307, 401 or a 200 of HTML, fix the route before you write a word of content.
2. If you ship docs, generate an llms.txt from the same source your docs are built from, and let a broken link fail the build.
3. Add markdown versions of your key pages and advertise them with a Link header. That is the part with a reader today.
4. Measure what you can actually measure. Start with how to measure AI visibility, then score your site's crawler rules, sitemap and descriptors with the free agent readiness check.