Quick answer
llms.txt is a Markdown file at the root of a domain that gives language models a curated map of the site's most useful pages, each with a one-line description. It is a discovery and comprehension aid, not an access control file — robots.txt governs permission, while llms.txt governs orientation.
What are the key takeaways?
- —llms.txt is a proposed convention, not a ratified standard, and support varies by engine — treat it as low-cost insurance rather than a ranking lever.
- —It never grants or denies crawler access; robots.txt is the only file that controls permission.
- —The format is Markdown: an H1 site name, a blockquote summary, then sections of linked pages with one-line descriptions.
- —The value is curation — telling a model which twenty pages matter, rather than leaving it to infer from two hundred.
- —Generate it from the same source of truth as your sitemap so the two can never drift apart.
What is llms.txt?
llms.txt is a Markdown file served at the root of a domain — /llms.txt — that describes the site to a language model in language a model can use directly. It typically opens with the site name, a short summary of what the site is for, and then a curated set of links grouped by section, each with a single sentence explaining what the page contains.
The motivation is context economy. A model reading a site cold has to infer structure from navigation, headings and internal links, all of which are designed for humans and full of noise. A curated file skips that inference and hands over the answer: here is what this site is, here are the pages that matter, here is what each one does.
It is important to be precise about its status. llms.txt is a community proposal that has been adopted by a growing number of sites and tools, not a formally ratified web standard, and not every engine consumes it. Publishing one takes an hour and costs nothing ongoing if you generate it programmatically, which is the correct way to think about the decision.
What llms.txt does not do
It does not control access. If you want to permit or block an AI crawler, that belongs in robots.txt, which is the file every serious agent actually checks for permission. Writing 'do not train on this' in llms.txt has no enforcement behind it whatsoever.
It does not replace your sitemap. A sitemap is a machine-readable inventory for crawlers, covering every indexable URL with metadata about priority and change frequency. llms.txt is deliberately the opposite: short, curated and opinionated, listing the pages you would point a knowledgeable colleague to rather than everything you publish.
And it is not a ranking factor. No engine has stated that the presence of the file improves selection. Its plausible benefit is comprehension: a model that understands your site's scope and terminology is more likely to reach for the right page of yours, and less likely to cite a weak page when a strong one exists.
How do you write a good llms.txt?
The conventional shape is an H1 with the site name, a blockquote holding a one-or-two-sentence summary, an optional paragraph of context, then H2 sections each containing a Markdown link list. Each link is followed by a colon and a single descriptive sentence. Keep it to the pages that genuinely represent the site — twenty to eighty entries is typical, and thousands defeat the purpose.
Write the descriptions for a reader with no context. 'Pricing' tells a model nothing; 'Pricing: per-seat and usage tiers with the exact formula used to calculate overage charges' tells it what question this page can answer. The descriptions do most of the work in the file, and writing them badly wastes the exercise.
Group by user intent rather than by your internal org chart. A model matching a prompt to a page benefits from sections like 'Calculators', 'Reference guides' and 'Policies' far more than from sections named after your content management taxonomy.
A working template
Begin with: an H1 containing your site name; a blockquote of one or two sentences saying what the site does and who it is for; a short note on how the content is produced and reviewed, which is a provenance signal worth including; then your sections.
For each section, list the canonical absolute URL of every page you want understood, followed by a colon and one sentence of description. Use absolute URLs, not relative paths — the file may be read in isolation from its host. Keep every entry pointing at a page that returns 200 and matches its own canonical, since dead or redirecting entries erode the file's usefulness quickly.
Close with an optional section for material you consider secondary, so a model that needs to economise knows what it can skip. Then automate generation from the same registry that produces your sitemap and navigation, and the file will stay correct without anyone remembering to maintain it. That is exactly how the file on this site is produced.
Is llms.txt worth publishing?
For most sites, yes, on a cost basis rather than a proven-return basis. If it is generated automatically the ongoing cost is zero, and the downside risk is nil since the file grants nothing and reveals nothing you have not already published. Even in the pessimistic case where no engine ever reads it, the exercise of choosing your twenty most important pages and describing each in one sentence is a useful audit in itself.
The case against is weaker than it looks. Some argue it exposes site structure, but a sitemap already does that more comprehensively. Others argue it is unproven, which is true and is precisely why it should be cheap rather than skipped — an hour of work against an uncertain but non-zero payoff is an easy trade.
Do not, however, let it substitute for the work that demonstrably matters: server-rendered answers, explicit robots rules for named AI agents, correct canonicals, schema and resolvable authorship. Publish llms.txt last, after those are in place.
What do people ask most about this topic?
Is llms.txt an official standard?
No. It is a community proposal that has seen meaningful voluntary adoption, not a ratified specification. Support varies by engine and tool, so treat it as inexpensive insurance rather than a requirement.
Does llms.txt control whether AI can use my content?
No. Permission is governed entirely by robots.txt and by the user-agent rules you write there. llms.txt only describes and orients; it has no access-control semantics and no enforcement.
Where should the file live?
At the root of the domain, served as plain text or Markdown at https://yourdomain.com/llms.txt with a 200 response. Some sites also publish an expanded llms-full.txt containing the full text of key pages.
How many pages should llms.txt list?
Enough to represent the site, not everything you publish. Twenty to eighty curated entries with genuinely descriptive one-liners is far more useful than an exhaustive dump, which simply recreates the sitemap without its machine-readable structure.
Should I write it by hand?
Write the descriptions by hand, then generate the file from the same page registry that produces your sitemap and navigation. Hand-maintained files drift within weeks; generated files stay correct indefinitely.
Written and reviewed by Jim Vernon, Editor, AI Intelligence International. Last reviewed 2026-08-24. Published by AI Answer Engine and checked against our editorial standards.