llms.txt

What the proposal is

llms.txt is a proposal by Jeremy Howard, published on llmstxt.org. The original version is dated 3 September 2024, and a revised version 2 followed in August 2026. The core of it fits in one sentence:

"We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content."

It is written for agents, not for search indexes. The proposal describes the intended use this way: "Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links." The file itself stays short, and "the detail lives behind the links, and is fetched only when needed."

Nothing in it is binding. It is a proposal, not a standard, and it describes itself as "open for community input", with a public GitHub repository for discussion.

What goes in the file

The format is Markdown in a fixed order. According to the proposal, a file contains:

• an H1 with the name of the project or site, "the only required section"
• a blockquote with a short summary
• optional paragraphs or lists with more detail, but no headings
• optional sections under H2 headings, each a list of links in the form [name](url), optionally followed by a colon and notes

A section headed "Optional" is, by convention, for links an agent can skip "when a shorter context is needed". The proposal's own mock-up:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

The links are meant to lead to something an agent can read easily. The proposal asks that they point "to LLM-friendly content, such as the markdown versions of pages", served at the page's own URL with .md appended or with the extension replaced by .md.

Two real files

The FastHTML project, the proposal's own example, publishes its file under its documentation path at fastht.ml/docs/llms.txt. Retrieved on 15 September 2026, it opens with an H1, a blockquote summary and a list of notes, followed by a "Docs" section of links to Markdown pages:

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications. […]

## Docs

- [FastHTML concise guide](https://www.fastht.ml/docs/ref/concise_guide.html.md): A brief overview of idiomatic FastHTML apps

Google's Gemini API documentation publishes one as well, at ai.google.dev/gemini-api/docs/llms.txt. It follows the outline loosely: an H1, a plain line of description where the proposal expects a blockquote, and a "Docs" list whose links end in .md.txt, a form the proposal does not describe.

What changed in version 2

The August 2026 revision keeps the format and adds rules drawn from how the file was being used.

A file can sit below the root. "The file can be placed at the site root, or at any path within it, covering the pages under that path." Where more than one file applies, "agents should use the most specific one."

Pages can point to it. Version 2 recommends standard link relations: rel="alternate" type="text/markdown" for a page's Markdown version and rel="describedby" for the llms.txt file that covers the page, either as HTML link elements or as an HTTP Link header.

Two URL forms for Markdown versions. Version 1 specified page.html.md. Version 2 also allows page.md.

The expansion tool is gone. According to the changes page, the context-expansion tooling "is no longer part of the proposal".

The author also writes that "thousands of sites publish an llms.txt file". That is his estimate, and the page gives no count or method behind it.

What Google says, in three places

Google Search does not need it. Google's documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The features are AI Overviews and AI Mode.

Chrome's Lighthouse checks for it. Among its agentic browsing audits, Lighthouse has one for llms.txt, which it calls "an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents." As documented, the audit flags a server error when the file is fetched, and treats a missing file like this: "If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment."

Google's API documentation publishes one. The Gemini API file described above is served from ai.google.dev.

The three do not contradict each other. Search says you do not need the file to appear in AI Overviews or AI Mode. Lighthouse treats it as optional and, as documented, flags only a server error. And one of Google's own developer sites publishes a file.

llms.txt is not robots.txt

The names invite confusion, and the proposal separates the two itself:

"robots.txt and llms.txt have different purposes. robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user."

An llms.txt file allows or blocks nothing. Access for AI crawlers is still set per crawler in robots.txt. It is not a sitemap either: the proposal notes that a sitemap generally covers documents "too large to fit in an LLM context window".

What you can check yourself

Whether anything reads the file can only be seen from your side. Requests for /llms.txt appear in server logs like any other request, with whatever user agent the client sends. Neither the proposal nor Google's documentation offers a report of use.

Before publishing one, check three things: it returns a normal response rather than a server error, which is what Lighthouse tests; its links lead to pages an agent can read; and it is short, because the proposal wants a file that "stays small enough to fit in context".

Common mistakes

Treating it as a Google ranking factor. Google's AI features documentation says no AI text files are needed.

Using it to block AI crawlers. It controls nothing. Blocking happens in robots.txt.

Linking to heavy HTML pages. The proposal asks for LLM-friendly content behind the links, such as Markdown versions of pages.

Putting the whole site into it. The file is meant to fit in context. The detail belongs behind the links.

Assuming it must sit at the root. Since version 2, a file can cover a subpath, such as a documentation folder.

Related terms

• robots.txt for AI Crawlers: the file that actually controls which crawlers may access a site.
• Context Window: the amount of text a model can take in at once, the limit llms.txt is designed around.
• AI Overview: Google's generated answer above classic results, for which Google says no AI text file is needed.
• AI Mode: Google's conversational search experience, covered by the same statement.
• Structured Data for LLMs: markup that describes a page's content and has to match what the page shows.