llms.txt: What It Is & How to Write One (Spec + Example)

llms.txt is a plain-markdown file served at your domain root (yoursite.com/llms.txt) that gives AI systems a concise, curated overview of your site: what it is, a one-paragraph summary, and organized links to your most important pages with short descriptions — so a language model can find and use your best content efficiently.

That’s the 44-word version. The motivation, per the proposal: language models have small context windows relative to a whole website, and HTML pages are cluttered with navigation, scripts and boilerplate that waste those tokens. A hand-curated markdown index lets an AI system skip the archaeology and go straight to the pages that matter, with your own framing of what each one is.

The proposal comes from Jeremy Howard of Answer.AI (September 2024) and is deliberately minimal. An llms.txt file is standard markdown with a fixed skeleton, parsed top to bottom:

1. 1An H1 with the site or project name. The only strictly required element.
2. 2A blockquote summary directly after — a short paragraph capturing what the site is and the key context needed to read everything else.
3. 3Optional free markdown (paragraphs, lists — no headings) for extra context about the site or how to interpret the files.
4. 4H2-delimited sections, each containing a markdown list of links in the form: link title, URL, then an optional colon-separated one-line description.
5. 5An optional “Optional” section — a specially-named H2 whose links can be skipped when an AI needs a shorter context; secondary material goes here.

The proposal also describes a companion convention: llms-full.txt, a single file containing the full expanded text of your key content (not just links) for systems that prefer to ingest everything in one fetch. It additionally suggests offering clean markdown versions of individual pages at the same URL with .md appended — same spirit, page level.

The keyword is curated. This is not a sitemap; dumping every URL defeats the purpose. Pick the pages you would hand a smart new hire on day one, and describe each in one honest line.

Here is a complete llms.txt for a fictional SaaS, following the spec end to end — H1, blockquote summary, context paragraph, H2 link sections with descriptions, and an Optional section:

# Acme Analytics

> Acme Analytics is a cookieless web analytics platform for
> B2B SaaS teams. It identifies the companies behind anonymous
> traffic and ties marketing channels to revenue, with a free
> tier and paid plans from $19/month.

Acme is GDPR-compliant by design: no cookies, no personal
data in the analytics layer. The docs below assume the
JavaScript snippet is installed.

## Docs

- [Quickstart](https://acme.example/docs/quickstart): install the
  tracking snippet and verify events in under five minutes
- [API reference](https://acme.example/docs/api): REST endpoints
  for events, goals and exports, with authentication examples
- [Integrations](https://acme.example/docs/integrations): Slack,
  Stripe and CRM setup guides

## Guides

- [Cookieless tracking explained](https://acme.example/guides/cookieless):
  how identification works without cookies, and its limits
- [B2B attribution basics](https://acme.example/guides/attribution):
  first-touch vs multi-touch, honestly compared

## Product

- [Pricing](https://acme.example/pricing): free tier, paid plans,
  and what gates each feature

## Optional

- [Changelog](https://acme.example/changelog): weekly release notes
- [About](https://acme.example/about): company background and team

Note the pattern in every link line: title, URL, colon, one-line description. The descriptions are doing real work — they’re what lets a model decide which page to fetch without fetching all of them.

If you’d rather not hand-write it, our free llms.txt generator builds a spec-compliant file from your site’s pages that you can edit and download.

Here is the part most articles skip: llms.txt is an emerging convention, not an adopted standard. It has real momentum — thousands of sites have published one, and several developer-tool companies generate them automatically — but the major AI vendors have not committed to reading the file, and none documents llms.txt as an input to its crawlers or answers. Claims that publishing one directly improves ChatGPT or Gemini citations are, today, unproven.

Treat it as one item in a broader generative engine optimization program — after crawler access and citable content, not instead of them.

The two files sit at the same level of your site and get confused constantly, but they answer opposite questions — robots.txt is about access, llms.txt is about comprehension:

They compose: robots.txt decides whether AI crawlers get in at all (see our note on CDN-level blocking — the common silent failure), and llms.txt helps whatever gets in understand what it found. Publishing llms.txt while blocking every AI crawler is decoration.

Three checks, in order:

1. 1Fetch it yourself. Open yoursite.com/llms.txt in a browser. It should return the raw markdown with a 200 status — not a 404, a redirect to your homepage, or an HTML-wrapped version.
2. 2Confirm AI crawlers can reach it. A perfect llms.txt behind a CDN rule that 403s AI user agents is invisible. Our AI crawler checker tests how your domain responds to the major AI bots in one click.
3. 3Keep it current. Stale links teach a model an outdated map of your site. Revisit the file when you ship major pages — quarterly is a fine cadence for most sites.

We practice what we describe: BusinessMCP serves its own file at businessmcp.com/llms.txt, generated and maintained alongside the guides it links. If you want to see a live production example beyond the fictional one above, that’s it.