Is llms.txt Worth It? Why It's Overrated (And What to Do Instead)

Let me be blunt: llms.txt is the most over-marketed file in the AEO world right now, and most of the people selling you on it are selling you a checkbox, not an outcome.

The idea sounds great — a tidy Markdown index that tells AI models exactly what your site is about. The reality is that the AI search engines you actually care about don't read it. I'm not saying it's useless. I'm saying it's been hyped into a priority it hasn't earned, and there are five things that will move your AI visibility far more, today. This page makes the contrarian case with real data, then tells you where to spend your effort instead.

What is llms.txt, exactly?

llms.txt is a proposed Markdown file you place at your domain root (/llms.txt) that gives language models a curated, plain-text index of your most important pages. It was proposed by Jeremy Howard, co-founder of Answer.AI, on September 3, 2024, and the spec lives at llmstxt.org.

The pitch is reasonable on paper. LLM context windows are limited, and converting messy HTML — nav bars, ads, JavaScript — into clean text is lossy. The proposal defines two files:

• /llms.txt — a short, human-curated list of key pages with one-line descriptions, like a sitemap written for a model.
• /llms-full.txt — an optional companion that dumps your full content into a single Markdown document.

Think of it as robots.txt's ambitious cousin. Where robots.txt controls access, llms.txt tries to provide context. It's a thoughtful idea. The problem is what happened next — which is to say, almost nothing.

Is llms.txt worth it for AI search visibility?

No. If your goal is showing up in ChatGPT, Perplexity, Google AI Overviews, or Gemini, llms.txt is decoration. The engines that decide whether you get cited do not consume it, and the data on this is no longer ambiguous.

Ahrefs analyzed server logs across 137,000 domains and found that 97% of existing llms.txt files attract no readers at all — human or bot (Ahrefs, June 2026). When a file was fetched, the readership wasn't AI search bots; the top requesters were GPTBot and Claude-Code, i.e. crawlers and coding tools, not the answer surfaces you're optimizing for.

Here's the part that kills the 'just in case' argument: Ahrefs found that for sites missing an llms.txt, the AI-bot share of those 404 requests was effectively zero. AI search bots don't even check whether the file exists. The humans probing for absent llms.txt files? Mostly SEOs snooping on competitors. The bots never go looking.

What do Google and the AI labs actually say about it?

They've told you, plainly, that it doesn't matter for search. Google has been the most direct, and OpenAI, Anthropic, and Perplexity have simply declined to document it as any kind of signal.

• Google lists llms.txt among the things site owners can ignore for generative AI search. In July 2025, Gary Illyes confirmed Google doesn't support it and isn't planning to.
• John Mueller (Google) put it on the record on Reddit: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." He compared it to the long-dead keywords meta tag — a signal everyone optimized that engines quietly ignored.
• OpenAI, Anthropic, Perplexity — none of their crawler or citation documentation asks you to publish llms.txt or treats it as a visibility requirement.

When the platforms themselves say 'we don't read this,' and your own server logs confirm it, that's not FUD. That's the answer.

Then why does everyone keep recommending it?

Because llms.txt is cheap to ship, easy to sell as a 'we're AI-ready' deliverable, and there's one legitimate use case that gets generalized into a universal one. The honest reason it persists is incentives, not evidence.

Three things keep the hype alive:

1. It's a frictionless checkbox. Agencies and AEO tools can 'add llms.txt' in an afternoon and show a tangible artifact. Real visibility work is slower and harder to invoice.
2. It pattern-matches to robots.txt. People assume a root-level text file 'for AI' must work the way robots.txt does. It doesn't — robots.txt is enforced by crawlers; llms.txt is a request crawlers ignore.
3. The coding-tool use case is real, and it gets laundered into a search claim. Which brings us to the one place llms.txt genuinely earns its keep.

Where llms.txt actually IS worth it: docs for AI coding agents

If you run a developer documentation site, ship it — this is the one scenario where llms.txt pulls real weight. AI coding agents actively look for it, and here the file does exactly what it promises.

Tools like Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, and Aider check for /llms.txt and /llms-full.txt when pointed at a docs site, using them to load clean, token-efficient context instead of scraping rendered HTML. That's why the named bots that do fetch these files skew toward GPTBot and Claude-Code rather than search crawlers (Ahrefs). It's also why early adopters were companies like Anthropic, Stripe, Cursor, and Mintlify — developer-tooling shops with docs that LLMs ingest directly.

Mueller framed it perfectly: llms.txt is "not done for search" — it's a "temporary crutch, perhaps to save some tokens" for coding tools parsing developer docs. So: documentation portal feeding agents? Yes, worth it. Marketing site chasing ChatGPT citations? No.

What to do instead: 5 higher-leverage AEO moves

Spend the hour you'd waste on llms.txt on things the engines actually consume. These are the levers that decide whether an AI answer cites you, in rough order of leverage.

1. Don't block the AI crawlers in robots.txt. This is the inverse of llms.txt and it's enforced. If GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, or Google-Extended are disallowed, no file on earth saves you. Audit this first.
2. Answer-first structure (BLUF). Open pages and sections with a direct 40–60 word answer, then elaborate. Models extract clean, self-contained answers — give them one to lift.
3. Schema.org structured data. FAQPage, Article, Organization, and Product markup are documented, parseable signals engines already use (schema.org). This is the 'machine-readable context' people think llms.txt provides — except it works.
4. Earn citations and mentions off-site. AI answers lean heavily on sources the model already trusts. Being referenced on Reddit, Wikipedia, and authoritative industry sites moves your inclusion rate more than any root-level file.
5. Measure what AI engines actually say about you. You can't optimize a black box you can't see. AEOeye's free AI visibility audit shows how ChatGPT, Perplexity, Google AI, Claude, and Gemini currently answer about your brand — so you fix what's broken instead of shipping files nobody reads.