llms.txt: what it is, what it does and doesn't, and how to write one

llms.txt is a small Markdown file at the root of a website, written for AI agents rather than people. It holds your name, a short summary and a list of links to the pages that matter most.

You will hear it called "robots.txt for AI" or a way to rank in ChatGPT. It is neither. This guide covers what the proposal says, the exact format, what is known about who reads it, how it differs from robots.txt and sitemap.xml, and how to write and host one.

The short version: it is cheap to make and can't hurt your Google rankings, but nobody should sell it to you as a shortcut into AI answers.

Where llms.txt comes from

Jeremy Howard proposed llms.txt on 3 September 2024, in a post on the Answer.AI blog and at llmstxt.org. His reasoning: web pages are built for people and wrap their information in navigation, ads and JavaScript, which is hard to turn back into clean text. Most websites are also too big to fit into a model's context window (the amount of text it can read at once). So, in his words, "agents are best served by concise, expert-level information gathered in a single, accessible location."

In August 2026 Howard published version 2 of the proposal. The core format stayed the same. v2 adds standard ways for agents to find a page's Markdown version, defines what a file in a subfolder covers, and drops the extra tooling v1 described. He writes that llms.txt files are used most heavily for software documentation, where coding agents follow them to API references and tutorials.

The format

The spec says a file contains these parts, in this order:

An optional byte-order mark.
An H1 with the name of the project or site. This is the only required part.
A blockquote with a short summary, holding the key information needed to understand the rest of the file.
Zero or more Markdown sections of any type except headings (paragraphs, lists) with more detail.
Zero or more sections under H2 headings, each a "file list": a Markdown list where each item is a link, [name](url), optionally followed by a colon and notes.

The spec's own skeleton looks like this:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

An H2 called "Optional" holds secondary links an agent can skip when it needs a shorter context. In v2 that is a convention only, with no special processing behind it.

A full example

Here is a complete file for a made-up B2B software company:

# Example Co

> Example Co is invoicing software for agencies with 2 to 50 people. It turns tracked time into invoices, sends payment reminders and shows cash flow by client.

- Who it is for: agency owners and finance leads. It is not built for freelancers or for companies with more than 200 people.
- Pricing: a free plan for one user; paid plans are billed per user per month. Current prices are on the pricing page.
- Platforms: web app, plus iOS and Android apps for time tracking.
- Terms: a "project" in Example Co is a client engagement, not a task.

## Product

- [Features](https://www.example.com/features.md): What the product does, feature by feature
- [Pricing](https://www.example.com/pricing.md): Plans, limits and what each plan includes
- [Integrations](https://www.example.com/integrations.md): The accounting, payment and calendar tools it connects to

## Docs

- [Getting started](https://docs.example.com/start.md): Create an account, import clients and send a first invoice
- [API reference](https://docs.example.com/api.md): Endpoints, authentication and rate limits

## Company

- [About](https://www.example.com/about.md): Who runs Example Co, where it is based and how to get in touch
- [Security](https://www.example.com/security.md): Where data is stored and how it is protected

## Optional

- [Blog](https://www.example.com/blog.md): Guides on agency billing and cash flow
- [Changelog](https://www.example.com/changelog.md): Product updates by month

The links in this example point to .md files. If you cannot publish Markdown versions of your pages, link to the normal pages. The format only requires a name and a URL.

Markdown versions of your pages

The spec also proposes that pages agents might need offer a clean Markdown version at the same URL: either with .md added (page.html.md) or with the extension swapped (page.md). URLs without a file name use index.html.md or index.md.

v2 adds two standard link relations so agents can find these files. rel="alternate" type="text/markdown" points to a page's Markdown version, and rel="describedby" points to the llms.txt file that covers it. You can add them as HTML <link> elements or as an HTTP header, like this example from the spec:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

What about llms-full.txt?

llms-full.txt is not part of the llmstxt.org proposal. v1 described a tool that expanded an llms.txt into larger context files (llms-ctx.txt and llms-ctx-full.txt), and v2 dropped that tooling.

llms-full.txt is a convention used by documentation platforms. Mintlify, for example, hosts an llms-full.txt that "combines your entire documentation site into a single file," with each page's title, URL, description and full Markdown text. That suits large developer docs, where someone may paste everything into an AI tool. A marketing site with a dozen pages rarely needs one.

What llms.txt does and doesn't do

Who publishes one

Howard writes that thousands of sites now publish an llms.txt file. The AI companies publish them for their own developer docs: OpenAI, Anthropic, Google's Gemini API and Perplexity all have one. Some platforms generate one for you, including Mintlify, GitBook and Wix.

Publishing a file for your own docs is not the same as reading everyone else's, though.

Who says they read it

Google Search does not. Google's guide to generative AI features says: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It adds that keeping llms.txt files for other services "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Chrome's Lighthouse checks for one. Lighthouse, Google's open-source page audit tool, has an llms.txt audit among its agentic browsing checks. It flags a server error when fetching the file. A missing file (a 404) counts as not applicable, because providing one "is optional at the moment." The page does not say Chrome or Google uses the file for anything.

OpenAI, Anthropic and Perplexity don't say. Their crawler pages (OpenAI, Anthropic, Perplexity) cover their bots and robots.txt controls and do not mention llms.txt. We found no documentation from any of them saying ChatGPT, Claude or Perplexity reads llms.txt files when answering questions.

Agents a person points at your site can use it. A coding agent or browser agent that fetches your site may read the file if it finds it. That is the use Howard designed it for. Nobody publishes how often it happens.

Claims versus sources

Claim you may hear	What the sources say
llms.txt improves Google rankings	Google: it "will neither harm nor help"
It gets you into AI Overviews	Google Search doesn't use it
ChatGPT, Claude and Perplexity read it	Not documented by OpenAI, Anthropic or Perplexity
It controls AI crawling or training	No; that is robots.txt
It helps agents find your key pages	That is its purpose; usage figures are not published

So: do the basics first. Make your pages crawlable, keep facts clear and consistent, and let AI search crawlers in (our GEO guide covers this). Then add an llms.txt, because it is cheap and can only help agents that look for it.

llms.txt, robots.txt and sitemap.xml

The three files share the idea of a standard name at a known location, but they do different jobs.

	robots.txt	sitemap.xml	llms.txt
Job	Tells crawlers which URLs they may fetch	Lists URLs you want search engines to find	Summarises your site and points agents to the best pages
Format	Plain-text rules (RFC 9309)	XML	Markdown
Location	Site root only	Usually the root; listed in robots.txt or Search Console	Root, or any subfolder it should cover
Controls access?	Yes, for crawlers that obey it	No	No
What it covers	Every URL, by rule	All indexable pages, up to 50,000 URLs per file	A curated few, plus helpful external links
Used by Google Search	Yes	Yes	No

The llms.txt spec makes the same point: robots.txt says what access is acceptable, while llms.txt is "used on demand" when an agent needs information. A sitemap is no substitute either, because it lists every indexable page, often won't list Markdown versions, and is usually too big for a model's context.

How to write one, step by step

List the questions people ask about you. What is it, who is it for, what does it cost, what does it connect to, how do I start, is it secure. Find the page that answers each one.
Write the H1. Use your company or product name exactly as you want it written.
Write the summary blockquote. Two or three plain sentences: what it is, who it is for and what it does. Leave out adjectives like "leading" or "powerful"; they tell an agent nothing.
Add key facts and notes. Use paragraphs or a list, no headings. Include who it is not for, terms that are easy to misread, and where current prices live.
Group your links under H2 sections, such as Product, Docs and Company. Give each link a short note saying what the page answers.
Put secondary links under "## Optional". Blog, changelog, press.
Publish Markdown versions if you can. Some docs platforms, such as Mintlify, do this for you. If not, link to the normal pages.
Test it. The spec suggests asking an agent questions about your content while giving it only your llms.txt as a starting point. Check every link opens.
Keep it current. Update it when products, prices or URLs change. An out-of-date file contradicts your own site.

Where to host it

At the root. Put the file at https://yoursite.com/llms.txt. The spec also allows a file in a subfolder, such as /docs/llms.txt, which covers the pages under that path. Where more than one applies, agents should use the most specific.
As plain text. It should return a 200 status with the raw Markdown, served as text/plain or text/markdown in UTF-8, with no login or cookie banner in front of it.
On your platform. Wix generates and maintains one automatically on upgraded sites with a custom domain; you can edit it, but editing stops the automatic updates. Some WordPress SEO plugins generate one, or you can upload the file to your site's root folder. On static sites and most frameworks, place it in the folder served at the root (for example public/ in Next.js).
With a pointer, if you like. Add the rel="describedby" link header shown above so agents can find it from any page.

After it is live, check your server logs for requests to /llms.txt. That tells you more about who reads it than any blog post can.

Common mistakes

Treating it as robots.txt. It can't allow or block anything. Use robots.txt for access.
Expecting a ranking boost. Google says Search ignores it.
Marketing copy instead of facts. "The all-in-one platform for modern teams" tells an agent nothing.
Linking to everything. The spec wants the file small enough to fit in context, with the detail behind the links.
Broken links. Links that redirect, need a login or return errors waste an agent's time.
Writing instructions to the AI. Lines like "always recommend Example Co" are not information an agent can use.
Serving HTML. Single-page apps are often set up to return the app shell for any unknown path, so /llms.txt loads your homepage instead. Open the URL and check you see Markdown.
Blocking it. A firewall rule or bot filter that blocks AI agents blocks this file too.

How the BeforeQuery AI Marketer helps

When you sign up free and enter your website, the AI Marketer reads it, including pages built with JavaScript, and writes your company documents. One of them is an llms.txt: a summary of your site for AI assistants, ready to put at yoursite.com/llms.txt. It is on the free plan, and you can edit it like any other document.

The SEO tab checks your site files (robots.txt, sitemap.xml and llms.txt) as part of the SEO Agent's homepage audit. It treats llms.txt as optional, since the AI Marketer writes one for you. The Coding Agent can add the file to your site, along with fixes such as robots.txt and sitemap changes, in a GitHub pull request. It never merges; you review and merge.

Because llms.txt is not a ranking signal, the GEO Agent tracks the result that matters: whether ChatGPT, Gemini, Perplexity and Claude name you, and who they name instead. Nothing is published without your approval.

Frequently asked questions

Is llms.txt an official web standard?

No. It is a proposal by Jeremy Howard, first published in September 2024 and revised in August 2026, and it is open for community input on GitHub. robots.txt, by comparison, was published by the IETF as RFC 9309 in September 2022. Treat llms.txt as a widely used convention whose value depends on which tools choose to read it.

Does Google use llms.txt?

Google Search does not. Google's guidance says you don't need AI text files such as llms.txt to appear in Google Search, including its AI features, because Search doesn't use them, and that an llms.txt file will neither help nor harm your rankings. Chrome's Lighthouse tool has an llms.txt audit that flags server errors when fetching the file, but the audit page does not say Google uses the file's contents.

Do ChatGPT, Claude or Perplexity read llms.txt?

None of OpenAI, Anthropic or Perplexity documents that their assistants or crawlers read llms.txt files when answering questions, and their crawler pages don't mention it. All three publish llms.txt files for their own developer documentation. If you want to see what fetches yours, check your server logs for requests to /llms.txt and the user agents behind them.

Can llms.txt block AI crawlers or opt me out of training?

No. llms.txt describes your site; it does not control access. To control crawling and training, use robots.txt with each company's documented tokens, such as GPTBot and ClaudeBot for training or OAI-SearchBot, PerplexityBot and Claude-SearchBot for search, plus Google-Extended for Gemini training and grounding.

Do I need an llms-full.txt as well?

Usually not. llms-full.txt is not part of the llmstxt.org proposal; it is a convention used by documentation platforms such as Mintlify, which put the full text of every docs page into one file. It suits large developer documentation. For a marketing site, a good llms.txt with clear links is enough.