Three files at the root of your site, three completely different jobs. robots.txt says who may fetch what. sitemap.xml lists every URL worth indexing. llms.txt hands a short curated reading list to an agent that is already inside. They are not substitutes and none of them can do another's work.
They also carry three different levels of authority. robots.txt is an IETF Standards Track document. sitemap.xml is a vendor agreement whose protocol page has not been revised since 2016. llms.txt is one person's proposal with no confirmed reader.
Key Takeaways
- robots.txt is standardised by RFC 9309, September 2022, Internet Standards Track.
- sitemap.xml is protocol 0.9 at sitemaps.org, last updated 21 November 2016, maintained by search vendors rather than a standards body.
- llms.txt is a proposal at llmstxt.org, v2 dated 10 August 2026, with no RFC and no vendor confirming it reads yours.
- The failure worth checking today: URLs listed in an llms.txt that robots.txt blocks.
The three at a glance
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Job | Access control | Enumeration | Curation |
| Answers | "May I fetch this?" | "What URLs exist?" | "What should I read first?" |
| Authority | RFC 9309, Standards Track | sitemaps.org 0.9, vendor agreement | Proposal, one author |
| Dated | Convention since 1994, RFC in 2022 | Protocol page revised Nov 2016 | Sept 2024, v2 Aug 2026 |
| Format | Plain text groups | XML | Markdown |
| Location | Origin root only | Anywhere, declared in robots.txt | Root or any subpath |
| Size limits | Parsers MUST read 500 KiB | 50,000 URLs, 50 MB per file | Small enough for a context window |
| Read by | Every compliant crawler | Google, Bing, others | No vendor has confirmed |
| Ignoring it | Non-compliant behaviour | Costs you nothing | Costs you nothing |
robots.txt: permission, and the only real standard here
RFC 9309, the Robots Exclusion Protocol, September 2022, Internet Standards Track, written by Martijn Koster with Gary Illyes, Henner Zeller and Lizzi Sassman of Google. A convention that had run on shared assumption since 1994 finally got exact parsing rules.
Two of those rules are worth knowing because they change how you write the file. Parsers must handle at least 500 kibibytes, so anything beyond that may be truncated mid-rule. And crawlers should not use a cached copy for more than 24 hours, which sets a realistic floor on how fast a change propagates. OpenAI's crawler documentation quotes roughly the same figure for its own systems.
What robots.txt does not do is the source of most of the damage in this area. It stops a compliant crawler from fetching. It does not remove anything already indexed, it does not reach crawlers you did not name, and it has no effect at all on a user-initiated fetch that the operator has decided is not crawling. Blocking a training crawler and blocking a search crawler are different acts with different consequences, and the two are constantly confused. Should you block AI crawlers works through that decision properly.
One more thing lives in this file now. Our own robots.txt carries a line like this:
User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /
That is Cloudflare's Content Signals Policy, released under CC0, adding search, ai-input and ai-train preferences to a robots group. Cloudflare states plainly that these express preferences and are not a technical countermeasure. So a real standard is now carrying a proposal inside it, and the proposal is honest about being unenforceable. Worth having, worth not mistaking for a lock.
sitemap.xml: enumeration, and a vendor agreement
The sitemaps protocol is version 0.90, published at sitemaps.org under a Creative Commons Attribution-ShareAlike licence, with "support from Google, Yahoo!, and Microsoft". The protocol page was last updated on 21 November 2016 and the site's front page in April 2020. There is no RFC and no IANA registration. That is what a de facto standard looks like: universal support, no governing body, no version bump in a decade.
The required tags are three: <urlset>, <url> and <loc>. Everything else, including <lastmod>, <changefreq> and <priority>, is optional and treated differently by different engines. The hard caps are 50,000 URLs and 50 MB per file, with sitemap index files subject to the same two numbers.
What a sitemap does not do is get you indexed. It tells a crawler that a URL exists and, through <lastmod>, roughly when it changed. Discovery is the cheap part of indexing. A sitemap full of thin pages does not become a sitemap full of ranking pages.
Its second job matters more than its first for this comparison: it is the only one of the three files that is machine-complete. It is supposed to list everything. That is exactly why it cannot do the llms.txt job, and the llms.txt spec says so directly: a sitemap covers documents that in aggregate are far too large for a context window and includes a great deal an agent does not need.
llms.txt: curation, and a proposal
The spec is at llmstxt.org, first published 3 September 2024 by Jeremy Howard and updated to v2 on 10 August 2026. The file is markdown: an H1 with the site name (the only required part), a blockquote summary, some free prose, then H2 sections holding lists of links with short notes.
Its status is the thing to be precise about. No RFC. No entry in the IANA well-known URI registry, and deliberately not a /.well-known/ file at all, for reasons covered in what /.well-known/ files agents look for. And no major AI product documents reading yours. Google is explicit in its generative AI guidance, updated 10 July 2026: you do not need machine readable files, AI text files, markup or Markdown to appear in Google Search, "as Google Search itself doesn't use them."
The longer version of that argument, including which vendors publish one for their own docs while saying nothing about consuming yours, is in llms.txt: what the spec says and who actually reads it.
The contradiction worth checking today
Because the three files are written by different people at different times, they drift into disagreeing, and one disagreement is worse than the others: an llms.txt that lists URLs your robots.txt blocks.
You have handed an agent a curated menu, signed with your site's name, in which some entries cannot be fetched by the crawler you allowed. The agent gets no helpful error. It gets a gap, and it fills the gap from somewhere else.
The check is mechanical. Take every URL out of your llms.txt, test each one against your robots.txt rules for the user agent you care about, and confirm each returns 200 to a signed-out request. Anything that fails is either a link to fix or a robots rule to loosen. Our llms.txt validator fetches every link in a file and reports the dead ones, and the agent readiness check looks at the crawler rules, the sitemap and the descriptors together rather than one file at a time.
Two smaller mismatches, both common: URLs in your sitemap that robots.txt disallows, which is a contradiction search engines will report back to you, and an llms.txt listing pages absent from the sitemap, which is usually fine and occasionally a sign the sitemap generator is missing a section.
Our own scoreboard, which is not clean
$ curl -sI https://seobuilder.tech/robots.txt | head -1
HTTP/2 200
$ curl -sI https://seobuilder.tech/sitemap.xml | head -1
HTTP/2 200
$ curl -sI https://seobuilder.tech/llms.txt | head -1
HTTP/2 307
Two out of three. The 307 sends a signed-out agent to our login page, because there is no llms.txt in the repository and our auth middleware catches every path that is not on an explicit allowlist. It looked fine to us for months, since a signed-in browser never sees the redirect.
That is the difference between the standard file and the proposed one, in one command. Nobody forgets robots.txt, because a search engine notices. Nothing notices a broken llms.txt.
Where to spend the hour
- robots.txt. Get the user agents right and know which ones govern training versus search. This is the only file here that can cost you traffic if you get it wrong.
- sitemap.xml. Generated, complete, under the caps, referenced from robots.txt, and containing only URLs that return 200.
- llms.txt. Only if you ship documentation and can generate it. Publish it and never look at it again and you have built a broken map with your name on it.
- Check for contradictions between all three, on a schedule, with a script rather than a browser tab.
None of this tells you whether you are being cited. For that you need to watch the answers themselves, which is what SEOBuilder does for Google AI Overviews and only for Google AI Overviews. How to measure AI visibility covers the method, including the parts you can do without paying anyone.