Is llms.txt Worth It? Cheap, but Don't Expect Citations

Create an llms.txt file only if it is cheap: an hour of work, or a file your platform generates. Do not expect it to get you recommended. Google says Search ignores it, the crawler documentation from OpenAI, Anthropic and Perplexity covers robots.txt and says nothing about it, and Ahrefs found that 97% of the files it measured received no requests in May 2026 (Ahrefs).

We publish one ourselves and still call it overrated. The exception is developer documentation, where coding agents do fetch it. For everyone else, five other moves matter more, listed at the end. For the file format and a build walkthrough, see what is llms.txt.

Should you create an llms.txt file?

Yes if it costs about an hour or nothing at all, no if it would displace crawl access or content work. Google says the file does not harm Search visibility, so the only real cost is your attention. Decide by who reads your site, not by hype.

We keep one at aeoeye.com/llms.txt because it was cheap to generate from our content index, not because we have seen it move a citation. Our robots.txt does more work: it allows GPTBot, ClaudeBot, PerplexityBot and Google-Extended explicitly.

Be wary of anyone who sells llms.txt as an AI visibility service. AI answers cite pages, not index files, so nobody can show you that the file produced a recommendation. A claim that cannot be disproved is easy to keep selling.

Does llms.txt help AI visibility?

No engine documents reading it to decide what to recommend. Google says its Search ignores the file, and the other vendors' crawler pages describe only robots.txt controls. Server logs agree: almost no AI search traffic asks for it.

• Google. Its generative AI guide (updated 10 July 2026) says you do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its AI features, because Search does not use them (Google Search Central). Its AI features page adds that a page must be indexed and snippet-eligible, with no special schema.org markup required (Google).
• Google's John Mueller. He wrote on Reddit that llms.txt is "comparable to the keywords meta tag" and that server logs show AI services don't check for it (Search Engine Journal, April 2025).
• OpenAI, Anthropic, Perplexity. Their pages explain robots.txt rules for OpenAI, Anthropic and Perplexity bots and describe no llms.txt signal. OpenAI and Perplexity mention the file only to point readers at their own documentation indexes.
• Server logs. Ahrefs, an SEO software vendor, studied 137,210 domains. Of the files, 97% got zero requests in May 2026. Of the requests that did arrive, 96% came from bots, and AI retrieval bots such as OAI-SearchBot and PerplexityBot made 1.1%. AI bots made no requests for files that did not exist.

Ahrefs' own verdict is that the cons outweigh the pros for AI search. It also warns that a fetch is not proof anything read or acted on the file.

How many sites use llms.txt?

Adoption is high among software companies and close to zero among news publishers. That pattern shows who sells to developers and follows SEO trends, not who gets cited. In AEOeye's crawler access study (October 2026, hub: AI crawler directory), 173 of 330 checkable sites (52.4%) served a valid llms.txt.

Two details matter more than the headline. Publishers are the sources AI engines lean on, and they skip the file. And only 28 of 384 robots.txt files (7.3%) point to llms.txt, so even adopters rarely announce it. Ahrefs found agents fetch the file when something tells them it exists.

Treat these as a snapshot. It is a convenience sample of established brands, fetched once on 3 October 2026, and a file counted only if it returned 200, was not HTML and began with a Markdown H1. We cannot tell hand-written files from generated ones (per-site data).

Across the wider web it is rarer. A July 2026 SIGCOMM CCR paper found llms.txt on about 49,600 of roughly 4 million domains, concentrated in tech and business (HPI summary).

Why does everyone recommend it anyway?

Because it is cheap to ship, easy to sell and impossible to disprove. No major AI platform has committed to reading it, yet the advice kept circulating. Four things keep it alive:

1. It is a visible deliverable. An afternoon produces a file you can show a client. Earning mentions on other sites takes months and is harder to invoice.
2. It sits next to robots.txt. People assume it works the same way. It does not: Ahrefs states that it controls nothing and blocks nothing.
3. One real use case got generalized. Ahrefs notes that the AI visibility framing came later, attached by the SEO industry on the speculation that AI platforms would reward the file.
4. The industry reads its own files. In Ahrefs' logs, 12% of fetches came from GEO and AEO tools, llms.txt checkers and researchers. Chrome's Lighthouse also audits the file, but marks it not applicable on a 404 because it is optional (Chrome for Developers).

The file also has downsides sold less loudly. Nearly 19% of llms.txt files in the SIGCOMM data link to content that robots.txt or ai.txt blocks, so they recommend pages the agent was told to avoid. Ahrefs found a research crawler probing llms.txt files for prompt injection, which is a reason to keep instruction-like text out.

Where is llms.txt actually useful?

On developer documentation that coding agents read. The proposal says the file is used most heavily for software documentation, where agents follow it to API references and tutorials (llmstxt.org). That is the one place we would call it good practice.

Three signals point the same way:

• Ahrefs ranked GPTBot first and Anthropic's Claude-Code second among named AI fetchers, ahead of every AI search and assistant bot.
• OpenAI, Anthropic and Perplexity each publish an llms.txt for their own developer docs (OpenAI, Anthropic, Perplexity). Perplexity's docs even carry a banner pointing AI agents to theirs.
• Ahrefs reports Mueller calling it not done for search, a stopgap for tools parsing developer documentation.

If you run docs, generate the file from your docs build, link Markdown versions of the pages, and test it by handing an agent only that file and asking questions. Steps are in how to add an llms.txt file, or use the free llms.txt generator.

What should you do instead?

Spend the hour on five things an AI engine actually has to get through before it can name you. They run in order, because each one is useless if the one before it fails.

1. Let the crawlers in. OpenAI says sites that disallow OAI-SearchBot do not appear in ChatGPT search answers (OpenAI). In our study few brands do this on purpose: 1 of 193 SaaS sites blocked any AI retrieval bot, against 27 of 38 publishers. But 24 of 88 e-commerce homepages (27.3%) challenged our self-identified crawler, so check your CDN and firewall logs. Start with should I block GPTBot and the AI robots.txt generator.
2. Stay indexed and snippet-eligible. Google's only technical requirement for AI Overviews and AI Mode is ordinary Search eligibility.
3. Write pages worth quoting. Open with the answer, one idea per section, one strong page per question. See what makes content quotable by AI. In October 2026 we merged 165 pages into 42 on that logic; it is our judgment, not a measured citation gain.
4. Get described accurately elsewhere. Engines build recommendations from what other sites say about you, and no file on your domain fixes that. See brand mentions.
5. Test real buyer questions. Ask each engine the questions your customers ask and read the answers. AEOeye's free audit asks ChatGPT three unbranded buyer questions and shows the answers in full. Daily monitoring across hundreds of prompts is what dedicated trackers are for (how to measure AI visibility).