The llms.txt proposal was introduced in September 2024 by Jeremy Howard of Answer.AI. It responds to a practical problem: AI systems work from limited context windows and often struggle to parse cluttered HTML pages, so a concise, curated map of a site's key content can help them use it more effectively.
How Is llms.txt Different From robots.txt?
They solve opposite problems. robots.txt tells crawlers what they may not access; it is a restriction file. llms.txt tells AI systems what is worth using and where to find it; it is a curation file. One draws a boundary, the other offers a guide. A site can use both, and they do not conflict.
What Goes in an llms.txt File?
The proposed format is a Markdown file at the site root. It typically opens with the site or project name and a short summary, then lists the most important pages as links with brief descriptions, often grouped into sections. An optional expanded version, sometimes published as llms-full.txt, includes the full text of key pages so a model can use them without additional fetches.
Because it is Markdown, it is readable by both people and machines, and it keeps the signal separated from the navigation, ads, and scripts that clutter a rendered HTML page.
Do AI Systems Actually Use llms.txt?
As of 2026 llms.txt is a community proposal, not an official standard adopted by the major AI companies, and support is not guaranteed. Some tools and platforms have begun generating and consuming it, but there is no assurance that any given model reads it. It should be treated as a low-cost, forward-looking addition, not a replacement for the fundamentals of crawlable, well-structured content.
Is llms.txt Worth Adding?
- It is inexpensive to create and maintain, especially for documentation-heavy sites.
- It does not harm anything, since it sits alongside existing files rather than replacing them.
- Its benefit is speculative until broader adoption, so it should not displace core technical and content work.
Frequently asked questions
What is llms.txt used for?+
llms.txt gives AI systems a curated, machine-readable guide, in Markdown, to a website's most important content. It helps models find and use the pages that matter instead of parsing cluttered HTML or crawling everything.
Is llms.txt the same as robots.txt?+
No. robots.txt restricts what crawlers may access, while llms.txt curates what AI systems should use and where to find it. One sets a boundary, the other offers a guide, and a site can use both without conflict.
Do AI models read llms.txt?+
As of 2026 it is a community proposal, not an official standard, and support is not guaranteed. Some tools generate and consume it, but no major model is guaranteed to read it, so it is best treated as a low-cost, forward-looking addition.