llms.txt

What llms.txt is meant to do

A site owner writes a llms.txt file in plain Markdown, typically a short site description plus links to the pages considered most important, and publishes it at the domain root, for example example.com/llms.txt. The proposal, put forward in 2024, imagines an AI system fetching this one file to get a curated map of a site instead of crawling and interpreting every page itself. The format borrows its name and its root-level placement from robots.txt, the older, genuinely standardized file that controls which crawlers may access which parts of a site, though the two files do very different jobs and carry very different weight with AI companies today. The proposal came from developer Jeremy Howard in September 2024 and spread quickly because it solved a problem framed in an intuitive way: large language models have a limited context window, and a short, curated file seemed like an efficient shortcut compared to crawling a full site. Thousands of sites, including many SaaS companies and documentation platforms, published one within the following year. The format itself has no required fields beyond a title and a short description, so the quality and usefulness of any given llms.txt file varies enormously from one site to the next.

How it works, and how it does not

Technically, llms.txt works like any other static text file: a crawler that chooses to fetch it can read a plain-language summary instead of parsing full HTML pages. Nothing forces any AI system to fetch it, though, and that is the open question the format has not resolved. In April 2025, Google's John Mueller said publicly that Google does not use llms.txt as a signal for crawling or indexing. No other major AI company has published confirmation that it reads the file either, and independent tests run through 2025 and into 2026 have not turned up a measurable citation benefit from adding one. The file can still exist on a site without causing any harm, it simply has not been shown to do the job its proposal describes. In practice, most sites that publish a llms.txt file also keep a working robots.txt and XML sitemap, so the file usually acts as an extra layer rather than the only way an AI system could find the content, whether or not that layer gets used. There is no equivalent of Google Search Console for llms.txt, so a site owner has no reliable way to confirm whether any AI system has actually fetched the file, only whether it exists and is reachable.

Why it matters for AI visibility

The gap between the idea and the confirmed evidence matters because time spent maintaining a speculative file is time not spent on signals AI systems demonstrably do use, like clean structured data, pages that answer a question directly, and access that is not blocked in robots.txt. AI visibility depends on what AI systems actually read and cite, not on what a site publishes for them in the hope that they will. llms.txt sits, for now, on the hopeful side of that line. That does not mean the idea is worthless in principle, only that its usefulness has not been demonstrated yet, and a format that no confirmed system reads cannot move a visibility number no matter how carefully it is written. Teams with limited time are better served spending it on the handful of signals that already have evidence behind them, and revisiting llms.txt later if an AI company ever confirms using it. The situation is genuinely unsettled rather than closed, which is different from it being proven not to matter.

Good practices

• Confirm your robots.txt does not block GPTBot, ClaudeBot, PerplexityBot or other AI crawlers before worrying about llms.txt at all.
• Treat structured data and clear, self-contained page content as the priority, since both have a documented effect on how AI systems use a page.
• Add a llms.txt file only after the above is in place, and only as a low-cost extra, not as a substitute for either.
• Re-check periodically whether any AI company has confirmed using the file; the situation has changed before and can change again.
• Measure the outcome that actually matters, whether your brand appears in AI answers, with an AI-visibility tool such as Truffle, rather than the presence of any single file. Truffle's LLM SEO guide covers where llms.txt fits among the signals that do have a confirmed effect.

Common mistakes

• Treating llms.txt as a ranking factor comparable to robots.txt or an XML sitemap, both of which have a confirmed, documented effect.
• Spending time maintaining a llms.txt file while AI crawlers remain accidentally blocked in robots.txt, which does have a measurable effect.
• Assuming that because the proposal was widely written about, every major AI company adopted it.
• Confusing llms.txt with structured data or schema markup, which are established signals AI systems do use.

Related terms

• GPTBot: OpenAI's crawler, one of the AI crawlers a site's robots.txt can allow or block.
• robots.txt for AI Crawlers: the standardized, genuinely effective way to control AI crawler access, unlike llms.txt.
• ClaudeBot: Anthropic's crawler, another AI crawler robots.txt rules apply to.
• PerplexityBot: Perplexity's crawler, a third AI crawler worth checking in robots.txt.
• Structured Data for LLMs: markup with a documented effect on how AI systems parse a page, unlike llms.txt.

Frequently asked questions