llms.txt is a plain-text Markdown file that a website places at its root (for example, example.com/llms.txt) to give AI tools a short, curated map of its most useful content. Think of it as a table of contents written for a machine reader instead of a human visitor.
The idea spread quickly through SEO and developer circles, but who actually reads the file is more nuanced than the hype suggests. Here is how the format works, what Google has said as of 2026, and when it is worth your time.
What is llms.txt?
llms.txt is a proposed standard, not an official web protocol. Jeremy Howard published the proposal on llmstxt.org in September 2024. It describes a Markdown file that gives AI agents a concise guide to a site, so they can find the right pages without crawling and parsing every HTML template, navigation menu and cookie banner.
The name borrows from robots.txt, but the two files do very different jobs. robots.txt tells crawlers what they may and may not access. llms.txt makes no statement about permissions at all. According to the proposal, it is meant to be used on demand, when an agent needs information while helping a user, which means it is aimed mainly at inference (answering questions in the moment) rather than model training.
The file is written for large language models (LLMs) and the agents built on them. Markdown is easy for those systems to read, and it is still readable for the humans who maintain it.
Why llms.txt matters (and why it might not)
The appeal is easy to understand. AI assistants have limited context windows, which is the amount of text they can consider at once. A long, cluttered web page wastes that space. A clean list of your best pages, each with a one-line description, could help an agent find your pricing page or documentation faster and summarize it more accurately.
The catch is adoption. Google has been unusually direct on this point. Its guide to optimizing for generative AI features in Search lists llms.txt in a "mythbusting" section and says Google Search does not use these files. Creating one will neither help nor hurt your visibility in Google Search, though the guide notes other services may use them. Google's separate page on AI features and your website also says you don't need new machine-readable files to appear in AI Overviews or AI Mode.
At the same time, another Google team has moved the other way. As Search Engine Journal reported, Lighthouse 13.3 added an Agentic Browsing category that checks for an llms.txt file. Chrome's documentation describes that Agentic Browsing category as experimental and based on proposed standards, and it is separate from SEO scoring. So the honest summary as of October 2026 is this: llms.txt is not a Google ranking or AI Overviews factor, but browser-based agents and some AI tools may find it useful.
How llms.txt works
The format is deliberately lightweight. A site owner writes one Markdown file, publishes it at a predictable path and keeps it current. Here is how the pieces fit together, based on the specification.
File location
The file is named llms.txt. It usually lives at the site root, but the proposal allows it at a subpath too, such as /docs/llms.txt. A file covers the URLs beneath its path, and agents are expected to use the most specific file that applies.
Required and optional sections
Only one element is required: an H1 heading with the name of the project or site. Everything else is optional but recommended, in this order:
- A blockquote with a short summary of the site.
- One or more paragraphs of extra detail, with no headings.
- H2 sections containing lists of links. Each entry is a Markdown link, optionally followed by a colon and a short note.
- An H2 section named "Optional" for secondary links that an agent can skip when it needs less context.
Markdown versions of pages
The proposal also suggests offering a clean Markdown copy of important pages at the same URL with .md added. It describes link relations that point agents to these files and back to the covering llms.txt. This part is optional, and it is where opinions differ most, since maintaining duplicate pages adds work.
How an agent uses it
An agent that supports the convention can fetch llms.txt first, scan the link list and pull only the pages relevant to the user's question. Whether a given AI product actually fetches the file is up to that product, and most don't say.
llms.txt in practice: a simple example
Imagine a hypothetical company that sells scheduling software for dental offices. Its llms.txt might look like this in plain words:
- An H1 with the company name.
- A one-sentence blockquote describing what the software does and who it is for.
- An H2 called "Product" linking to the features page, the pricing page and the integrations list, each with a short note.
- An H2 called "Help" linking to setup guides and the HIPAA compliance page.
- An "Optional" H2 linking to the company blog and press page.
Real examples are common on developer documentation sites. Anthropic publishes an llms.txt for its developer docs with an H1, H2 sections and lists of linked pages with short descriptions. Documentation is a natural fit because coding assistants often need to look up exact API details.
How to create and maintain an llms.txt file
If you decide to publish one, keep it small, accurate and easy to update. These steps cover most sites.
- Pick your priority pages. Choose the 10 to 40 pages that best explain what you do: core service or product pages, pricing, key guides, contact and about pages. Skip thin or outdated content.
- Write a factual summary. Use one or two plain sentences in the blockquote. State what the business is, who it serves and where. Avoid slogans.
- Group links logically. Use a few H2 sections that mirror how a customer thinks, such as Services, Pricing, Guides and Company.
- Add short notes. After each link, write a brief description so an agent knows what the page answers.
- Publish at the root and return a clean 200 status. Serve it as plain text. Make sure your server doesn't return errors for the file, since the Lighthouse audit flags server errors.
- Keep it in sync. Add a review step to your publishing process so new pages are added and retired pages are removed.
- Fix the fundamentals first. Clean indexing, clear headings and accurate schema markup matter far more for search than any text file.
Common mistakes and misconceptions
"llms.txt will get me into AI Overviews." Google says otherwise. Its documentation states there are no special files or markup required for AI features, and pages simply need to be indexed and eligible to show a snippet.
"It blocks AI crawlers." It doesn't. llms.txt carries no access rules. If you want to control crawling, use robots.txt and the user agents each company documents.
"More is better." Dumping your full sitemap into llms.txt defeats the purpose. The value is in curation.
"Set it and forget it." A file that points to broken or outdated pages sends agents to the wrong information, which is worse than having no file.
llms.txt vs. robots.txt and sitemap.xml
These three files are often confused. robots.txt tells crawlers which paths they may access. sitemap.xml lists URLs you want search engines to discover. llms.txt is a curated, human-readable guide meant to help AI agents understand your most important content.
In short, robots.txt is about permission, sitemap.xml is about discovery, and llms.txt is about comprehension. You should have the first two in good shape before thinking about the third.
How Butter approaches llms.txt
At Butter, we treat llms.txt as a low-cost, optional extra, not a growth lever. We tell clients plainly what Google has said: it doesn't use the file for Search or AI Overviews. We also know that other agents and tools may read it, and publishing a tidy one takes very little effort once the site itself is in order.
So our order of operations is fundamentals first. On our GEO plan, monthly AI prompt testing and indexing strategy show us how assistants actually describe a client today, and schema markup guidance and knowledge graph submissions help make the brand's facts clear and consistent. If a client already has clean, crawlable pages and wants an llms.txt, we help them write a curated one and keep it current in their on-page recommendations.
After years in SaaS growth and agency SEO, we have seen many "must-have" files come and go, so we would rather measure real AI visibility than chase a checklist item. You can see how this fits into our GEO service.
Frequently asked questions
Does Google use llms.txt?
No. Google's guide to generative AI features in Search says Google Search does not use llms.txt and that creating one won't help or hurt your Search visibility. A separate Lighthouse audit in Chrome checks for the file, but it is experimental and not tied to rankings.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they are allowed to access. llms.txt is a curated Markdown guide to your most useful content and contains no access rules, so it cannot block or allow any bot.
Where should I put my llms.txt file?
Place it at the root of your domain, such as example.com/llms.txt, and serve it as plain text with a 200 status. The proposal also allows files in subfolders like /docs/llms.txt to cover a specific section of a site.
Should a small business bother with llms.txt?
It is optional. If your site is already crawlable, well organized and accurate, a short llms.txt takes little time and may help some AI agents. If those basics aren't in place yet, fix them first, because they matter much more.
Sources and further reading
- The /llms.txt file proposal, llmstxt.org
- Optimizing for generative AI features on Google Search, Google Search Central
- AI features and your website, Google Search Central
- Lighthouse agentic browsing, Chrome for Developers
- Google's llms.txt guidance depends on which product you ask, Search Engine Journal