How to Add an llms.txt File: Create, Host and Validate It (2026)

Four things: your site name, a one-sentence summary, optional context, and short lists of links to the pages you want an agent to read first. The spec requires only the name; the rest is curation.

Link to pages that answer what a buyer or an agent would ask: product and docs, pricing, shipping and refund terms, security, and your two or three best guides. Give each link a plain one-line note on what the page contains. The spec also recommends clean Markdown copies of pages (page.md) so agents read text instead of navigation and scripts.

Before publishing, ask of every URL: would I want an AI to treat this as my canonical answer? Perfect formatting can still point at retired pages. Background: what llms.txt is.

A Markdown file with sections in a fixed order: an H1 (required), a blockquote summary, optional paragraphs without headings, then H2 sections that each hold a list of links (llmstxt.org). Each link is written as - [name](url): optional note. This example uses a made-up company:

# Acme Analytics

> Privacy-first web analytics for SaaS teams. Free tier, EU hosting.

Acme tracks events without cookies. Pricing is per monthly event volume.

## Docs

- [Quickstart](https://acme.example/docs/quickstart): install the tracker and send a first event
- [API reference](https://acme.example/docs/api): REST endpoints, limits, errors

## Pricing and policies

- [Pricing](https://acme.example/pricing): plans and event limits
- [Refunds](https://acme.example/refunds): refund terms

## Optional

- [Changelog](https://acme.example/changelog): release history

Rules that trip people up:

• One H1, no headings before the first H2.
• ## Optional is a convention. Version 2 dropped the context-expansion tool that gave it mechanical meaning, so it now only signals lower priority (spec changes).
• Subfolders work. /docs/llms.txt covers the pages under /docs, and the most specific file wins.
• llms-full.txt is not in the spec. Docs platforms such as Mintlify generate it as the whole documentation in one file.

Our free llms.txt generator builds this layout from a form. It does not crawl your site, so you choose the pages.

Save the file as UTF-8 text named llms.txt and make it load at yourdomain.com/llms.txt. On several platforms you change a setting instead of hosting a file.

Ahrefs concludes agents fetch the file when directed to it, and found zero AI-bot requests for missing files. So point to it: Perplexity's docs open with a note telling agents to fetch their /llms.txt, and the spec offers a Link header with rel="describedby".

robots.txt has no llms.txt directive (RFC 9309 defines none), though 28 of 384 robots.txt files (7.3%) in our study mention it.

Fetch it the way a bot would and check four things: HTTP 200, not HTML, still at /llms.txt after redirects, and a first line starting with "# ". That is our study's rule; it takes two minutes.

1. Status, type, final URL. Run curl -sL -o /dev/null -w "%{http_code} %{content_type} %{url_effective}\n" https://yourdomain.com/llms.txt. Pass: 200, text/plain or text/markdown, and a final URL ending in /llms.txt. OpenAI's file is text/plain; Google's Gemini API docs file is text/markdown.
2. First line. Run curl -sL https://yourdomain.com/llms.txt | head -n 3. The first line must be "# " plus your name. An html tag means you serve a page, not a file.
3. Soft 404. A soft 404 is an error page that returns 200. Single-page apps are a classic cause: with client-side routing, meaningful status codes can be impractical. With no file, return a real 404.
4. Bot view. Repeat step 1 with a bot-style header, for example -A "Mozilla/5.0 (compatible; TestBot/1.0)". A 403 or challenge page means a bot rule is in the way: 46 of the 376 sites whose robots.txt let our crawler request /llms.txt answered with a block or error. A spoofed header is imperfect, so check your CDN's bot log too.
5. Links. Every link should return 200 and point to a page you control.
6. Ask an agent. The spec suggests asking an assistant questions about your business with only your llms.txt as the starting point. If it cannot find your pricing, your notes are too vague.

Chrome's Lighthouse has an experimental llms.txt audit that needs Chrome 150 or later. Its documentation describes only two outcomes, a flagged server error or not applicable on a 404, so do not count on it to catch a broken format. Afterwards, search your access logs for /llms.txt.

Leave out anything you would not want an agent to quote or follow: a dump of every URL, thin or retired pages, login and legal boilerplate, and anything written as instructions. Short and accurate beats long and stale.

• A sitemap mirror. The median valid file in our study was about 9,200 characters; 17 of 173 ran past 50,000.
• Instruction-shaped text. Agents are built to trust the file. Ahrefs advises plain links and descriptions only, links to resources you control, restricted edit access and review of anything a platform generates.
• Stale links. A file pointing at deleted pages is confidently wrong, not merely absent. If a check fails, regenerate rather than hand-patch: one missing bracket breaks a link silently.

We break the first rule ourselves. aeoeye.com's file is generated from our content collections and listed every article, guide, answer and glossary term when the study fetched it: about 92,000 characters, longer than all but 12 of the 173 valid files. We keep it because it costs nothing, not because it moves citations.

Worth an hour, not a project. Google's AI optimization guide says Search does not use these files and that creating one will neither harm nor help your visibility. Ahrefs found that 97% of the roughly 38,000 valid files it examined (from 137,210 domains) got no requests at all in May 2026.

Ahrefs notes its sample skews technical and that a fetch is not a read. Where the file can pay is developer documentation: GPTBot and Claude-Code were the top named AI fetchers in its logs, and the spec's author notes that OpenAI, Anthropic and Gemini publish an llms.txt for their own developer docs. If coding agents use your docs, ship the file with Markdown pages.

If you run a store or service business hoping ChatGPT names you, do the basics first: let AI search crawlers in (crawler directory), then measure whether AI names you with the free audit. Our full argument is in why llms.txt is overrated.