Llms.txt Explained: Should You Add One to Guide AI Crawlers on Your Site?

llms.txt is a proposed standard — a plain markdown file at your site’s root that hands AI systems a curated map of your best content — and the honest 2026 answer is: add one if it takes you twenty minutes, but expect nothing from it yet, because no major AI provider has confirmed actually reading the file. That is the entire llms.txt SEO story in one sentence. Everything else is detail: what the file looks like, why smart people proposed it, why adoption is lopsided, and how to add one without wasting an afternoon on a tactic nobody can measure.

Key takeaways

• llms.txt is a markdown file at /llms.txt proposed by Jeremy Howard (Answer.AI) in September 2024. It summarizes your site and links to your most important pages so language models can skip the HTML clutter.
• It is an invitation, not a control. robots.txt tells crawlers what they may not fetch; llms.txt suggests what an AI reading your site should look at first. The two files solve opposite problems.
• Publishing is widespread — documentation platforms like Mintlify generate the file automatically, which is why so many developer docs have one — but consumption is unconfirmed. Google’s John Mueller has said no AI system he knows of uses it, likening it to the keywords meta tag.
• It is not a ranking factor, it does not affect AI Overviews, and it will not move traffic. The case for adding one is cost, not payoff: it is nearly free insurance in case the standard catches on.
• If your content itself is thin, no index file rescues it. Models cite pages worth citing; llms.txt just makes them marginally easier to find.

What llms.txt actually is

The specification, published at llmstxt.org, is short enough to read over coffee. An llms.txt file is ordinary markdown with a fixed shape: an H1 with your site or project name, a blockquote giving a one-to-two-sentence summary, optionally a few paragraphs of context, then H2-labeled sections containing link lists — each entry a page title, a URL, and a one-line description of what is there. An optional final section called “Optional” marks the links a model can skip when context is tight.

The reasoning behind it is genuinely sound. Language models have context windows, and your website — with its navigation, cookie banners, footers, scripts, and layout markup — is a hostile format for them. A model that wants to answer a question about your product either burns tokens parsing all of that or gives up and answers from stale training data. A hand-curated markdown map says: here is what this site is, here are the twelve pages that matter, start there.

The spec also describes a bigger sibling, llms-full.txt, which inlines the complete text of your key pages into one giant markdown document. Documentation platforms took to this quickly — Mintlify added automatic generation in late 2024, which instantly gave thousands of developer-docs sites both files, and is the main reason llms.txt feels ubiquitous in the software world while remaining rare everywhere else.

The llms.txt SEO question: does anyone actually read it?

Here is where honesty matters more than enthusiasm. The llms.txt SEO pitch assumes step two of a two-step process: you publish the file, and AI systems consume it. Step one is booming. Step two is where the evidence runs out.

No major provider — not OpenAI, not Anthropic, not Google, not Perplexity — has documented that its crawlers or assistants fetch and use llms.txt. Google’s John Mueller addressed it directly in a mid-2025 forum comment: no AI system he was aware of used the file, and he compared it to the keywords meta tag, the classic example of site owners describing themselves to machines that stopped listening. Site owners who watch their server logs report the same pattern — AI crawlers hitting HTML pages constantly and the llms.txt file rarely, mostly when a curious human or an SEO tool fetches it.

Set against that, the counterargument is real but modest: standards sometimes work this way. Sitemaps existed before every engine honored them. The file costs nothing to serve, and if a provider flips the switch next year, sites that already publish one are ahead by exactly one afternoon of work. Just do not confuse that option value with a tactic. If someone sells you an “llms.txt optimization service,” keep your wallet in your pocket.

What demonstrably does influence whether assistants cite you is the substance the file merely points at — clear answers, quotable statements, real expertise on pages that rank. That is the territory of generative engine optimization, and it is where the measurable wins live. Research on how LLMs choose which sources to cite keeps landing on the same conclusion: models reward content quality and retrievability, not metadata gestures.

llms.txt vs robots.txt: opposite jobs

People conflate these constantly, and the confusion causes real mistakes.

• robots.txt is access control. It tells named crawlers which paths they may not fetch. It is where you decide whether you should block AI crawlers in robots.txt — a decision with actual consequences for training-data use and AI-answer visibility.
• llms.txt is hospitality. It assumes the AI is welcome and offers it a reading list. It cannot block anything, enforce anything, or protect anything.
• sitemap.xml is inventory for search indexing — every URL you want crawled, no prioritization, no descriptions, built for a pipeline that already parses HTML at scale.

A coherent setup uses robots.txt to draw the lines you actually care about, a sitemap for the index, and — if you like — llms.txt as the curated front door for whatever reads markdown someday.

How to add one in twenty minutes

1. Pick ten to fifteen pages that define you. Product pages, cornerstone guides, docs, pricing, about. The pages you would show a new employee on day one.
2. Write the header. # YourSite, then a blockquote: one or two plain sentences saying what the site does and for whom. No slogans — models do not convert on slogans.
3. Group links under H2 sections. Something like ## Guides, ## Product, ## Company. Each entry: - Page title: one line on what the page answers. The description line is the part worth writing well; it is the anchor text of this format.
4. Add an ## Optional section for secondary links a model can drop when context is short.
5. Serve it at your root as plain text at /llms.txt, and add a calendar note to refresh it when your key pages change. A stale map pointing at dead URLs is worse than no map.

One quality note before you ship it: the file advertises whatever it links to. If the pages behind those links read like templated filler, you have just given machines a shortlist of your weakest work. It is worth a pass to check how your writing reads on the pages you are about to showcase, the same way you would proofread anything you put in front of an important reader.

Frequently Asked Questions

What is an llms.txt file? It is a plain markdown file placed at your site’s root (yoursite.com/llms.txt) that gives AI systems a curated summary of your site: an H1 with the site name, a short blockquote describing what you do, and sections of links to your most important pages with one-line descriptions. The idea is to hand a language model the clean, prioritized version of your site instead of forcing it to parse navigation menus, scripts, and boilerplate HTML.

Is llms.txt an official web standard? No. It is a proposal, published in September 2024 by Jeremy Howard of Answer.AI, the researcher who co-founded fast.ai. Nothing about it has been ratified by a standards body, and no major AI provider has committed to reading the file. Adoption so far is one-sided: many sites publish one, especially developer documentation platforms, but publishing is not the same as consumption.

Does llms.txt improve Google rankings? There is no evidence that it does, and Google has been fairly direct about this. John Mueller commented publicly in mid-2025 that no AI system he was aware of used llms.txt, comparing it to the old keywords meta tag — a place where site owners describe themselves that machines mostly ignore. Googlebot does not need the file, and AI Overviews are built from Google’s ordinary index, not from llms.txt.

What is the difference between llms.txt and llms-full.txt? llms.txt is a map: a short markdown index of links with descriptions, designed to fit comfortably in a model’s context window. llms-full.txt is the territory: the actual content of your key pages concatenated into one large markdown document, so a model can ingest everything in a single fetch. Documentation platforms often generate both automatically. For most content sites, the small index file is the sensible starting point.

How do I create an llms.txt file? Write a markdown file by hand or generate one: start with an H1 naming your site, add a blockquote summarizing what it offers in one or two sentences, then add H2 sections (for example Docs, Guides, About) containing bulleted links in the format Page title: one-line description. Save it as llms.txt, serve it from your domain root as plain text, and keep it updated when your key pages change. Ten to twenty minutes covers most sites.

Bottom line

llms.txt is a reasonable idea waiting for a reader. The format is sensible, the cost is trivial, and the benefit is currently a bet on adoption that has not happened — no confirmed consumer, no ranking effect, no traffic story. Spend the twenty minutes, publish the file, and then put your actual energy where AI visibility is really decided: pages worth citing, structured clearly enough to quote. There is more on AI search and SEO in the cluster, and the tools for auditing how your content reads at scale are on the pricing page.