What is llms.txt and do you actually need one?

llms.txt is a proposed plain-text manifest you publish at /llms.txt — like robots.txt, but instead of telling crawlers what to skip, it gives LLMs a curated, link-rich tour of your site. Hand them a map and they're more likely to cite the right pages.

What it is

A markdown file at the root of your domain. First section: a single-sentence summary of what your site is. Then short, scannable sections grouped by content type (Docs, Blog, Pricing, Changelog) with bullet links and one-line descriptions. The format is deliberately minimal so LLM context windows can ingest it cheaply.

# Acme Cloud

> Acme Cloud is a managed Postgres provider for teams that hate ops. Free tier, point-in-time recovery, branching, 30s provisioning.

## Docs
- [Quickstart](https://acme.cloud/docs/quickstart): Provision your first database in 30 seconds.
- [Branching](https://acme.cloud/docs/branching): Per-PR database branches via API.
- [Backups](https://acme.cloud/docs/backups): Point-in-time recovery up to 14 days.

## Pricing
- [Pricing](https://acme.cloud/pricing): Free up to 500MB. Pro at $19/mo.

## Blog
- [Why we built branching](https://acme.cloud/blog/branching): The engineering case for ephemeral DB branches.

What it is NOT

• Not an access-control file — it doesn't block or allow bots. Use robots.txt for that.
• Not an indexing signal for Google Search. Googlebot ignores it.
• Not a sitemap. Sitemaps are exhaustive lists for crawlers. llms.txt is a curated tour for LLM context.
• Not yet a ratified standard — it's a proposal at llmstxt.org with growing voluntary adoption.

Who's using it

As of 2026, hundreds of dev-tool sites ship one — Anthropic, Mintlify, Vercel docs, Hono, Drizzle, Pydantic, FastHTML, and most modern docs SaaSes. Adoption is concentrated where LLM users frequently ask about products (dev tools, APIs, libraries, documentation-heavy sites). Less common on lifestyle / e-commerce sites, but the same logic applies.

Does it actually move the needle?

Honest answer: nobody's published a controlled study yet. What we do know is that when LLMs cite a docs site, they're more likely to cite the exact deep link if there's an llms.txt that names that page and summarizes it in one line. Anecdotally, dev-tool teams report cleaner ChatGPT and Claude citations after publishing one. It's cheap to ship and the downside is zero.

How to write a good one

1. Start with a single-sentence pitch in a blockquote at the top. Treat it like a tweet.
2. Group pages by intent: Docs, API, Pricing, Blog, Changelog, About. Not by URL structure.
3. Each link gets one line. Title, URL, one-sentence description. Trim aggressively.
4. Order sections by how often you'd want them cited. Quickstart first, deep API reference second, opinionated blog post third.
5. Cap the file at ~500 lines. LLM context windows are generous now but readability still matters.
6. Refresh it when you ship something major. Stale llms.txt files mislead.

llms-full.txt — the longer cousin

Some sites also publish /llms-full.txt with the full markdown body of every linked page concatenated. Useful for technical docs where you want LLMs to ingest the entire content set in one fetch. Skip it for marketing-heavy sites — the noise hurts more than the depth helps.

Pair with the AI bot auditor

Before you publish, run your domain through our AI bot auditor to see which crawlers your robots.txt currently allows. If you're blocking the on-demand fetchers (ChatGPT-User, Claude-Web, PerplexityBot), your llms.txt won't get read. Fix the access policy first, then ship the manifest.