Does llms.txt Work? What AI Vendors Document

On this page, 8 sections

I could not find a single AI vendor that documents fetching llms.txt from your site to decide what it cites or ranks. Google’s AI features guide says you do not need to create new machine readable files, AI text files or markup to appear in those features.

That does not make the file useless, and it does not make it proven. It means the claim “llms.txt improves AI visibility” has no vendor documentation behind it that I could find, and that the people who benefit most are agents reading documentation, not search engines ranking your blog.

I checked this on 11 October 2026 by reading three vendors’ crawler pages and Google’s AI features guide, then fetching the llms.txt files those vendors publish for themselves.

What each vendor says about it

The first test is the plain one. Do the pages that tell site owners how each AI crawler behaves mention llms.txt? I read four: three vendor crawler pages and Google’s AI features guide.

Google, AI features and your website: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” There is also no special schema.org markup needed, per the same page.
OpenAI, crawler overview: the page describes GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. It does not describe any of them fetching llms.txt. The one mention of the file on the page is a banner pointing to OpenAI’s own documentation index.
Anthropic, crawler help article: three bots described, zero mentions of llms.txt.
Perplexity, crawler documentation: two agents described. Again the one mention is a banner pointing to its own documentation index.

I got these counts by searching the text of each page I fetched for “llms”. Zero of the four pages say their crawler requests llms.txt from your site. That is what the sources say. It is not proof that none of them ever fetches it.

The vendors publish the file for themselves

There is a reading of this that favours the file. The same vendors publish one for their own docs. I fetched these on 11 October 2026, following redirects:

docs.anthropic.com/llms.txt: 82,696 bytes.
developers.openai.com/llms.txt: 6,201 bytes.
docs.perplexity.ai/llms.txt: 37,565 bytes.

The spec site, llmstxt.org, also says the AI labs publish one for their developer docs and names OpenAI, Anthropic and Gemini. That is a statement from the spec’s author and I checked OpenAI’s and Anthropic’s, not Gemini’s.

What this shows: the file is used where a developer points an agent at a documentation site. It does not show that a consumer assistant reads yours before answering a question about your business.

What the spec claims, and what I could not check

The v2 text of the proposal says that thousands of sites publish an llms.txt, that documentation platforms generate one automatically, and that Chrome’s Lighthouse audits sites for one as part of its agentic browsing checks. I read that on 11 October 2026 on llmstxt.org. I did not verify the Lighthouse claim or the count of sites.

Note that a Lighthouse audit checking for the file is a statement about a lint rule, not about whether an AI system uses the file. Those are different claims, and the page does not conflate them, but readers often do.

For a longer look at the format, the plugins that generate it and where my own implementation fell short of the spec, see llms.txt for WordPress: the spec and what adoption actually looks like.

What I could not verify: who actually requests it

The question that would settle this is which crawlers request /llms.txt on a real site. Server logs answer it.

I could not answer it for adityaarsharma.com in this session. The site is behind Cloudflare (the response header reads server: cloudflare), and I did not have its raw access logs. So I have no log evidence and I am not going to fabricate any.

You can run the check yourself on any site with access logs:

grep -E "GET /llms(-full)?\.txt" access.log \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn | head -20

That prints each user agent that requested the file and how often. Then check the source IPs of any claimed AI bot against the vendor’s published ranges. GPTBot vs OAI-SearchBot vs ChatGPT-User shows how, using OpenAI’s JSON lists.

If the list of user agents is only browsers and generic fetchers, that tells you something. If you do see named AI bots, you have evidence the documentation does not give.

Where it is closer to a sitemap than to robots.txt

The comparison that holds up is with sitemaps, not with robots.txt. robots.txt is a rule file that crawlers have agreed to read. A sitemap is a helpful list that a crawler may use or ignore. llms.txt is in the second category, with one extra problem: no vendor I read has said it reads it.

I wrote about how a sitemap actually gets served on WordPress in WordPress sitemaps: core versus plugin. The pattern there is the same as here: check what the plugin does, check what the consumer says it reads, and do not credit a file with an effect nobody has documented.

What my own file looks like today

I fetched my own files on 11 October 2026:

adityaarsharma.com/llms.txt: 37,394 bytes, HTTP 200.
adityaarsharma.com/llms-full.txt: 1,238,974 bytes.

On 2 September 2026 I measured the full file at 672,228 bytes, in the post linked above. The file is almost twice that size now. A file that large defeats the purpose of a summary for a model with a limited context, which is a point I made in that earlier post. The spec’s own reasoning is that every wasted token costs time and money.

I run these files, so I am not telling you to remove yours. I am telling you not to measure their value by their size.

Questions people ask

Does llms.txt help SEO?

Google says you do not need AI text files to appear in its AI features. I found no vendor documentation that says otherwise.

Does ChatGPT read llms.txt?

OpenAI’s crawler page does not say it fetches llms.txt from your site. OpenAI does publish one for its own documentation.

Is it worth adding to a WordPress site?

It costs little to generate. If your site is documentation that developers feed to coding agents, there is a plausible reason for it. For a blog, I have no evidence of a benefit.

How do I know if anyone reads mine?

Search your access logs for requests to the path, grouped by user agent, using the command above.

Do this today

Run the grep above on your own access logs for the last 30 days. Write down the user agents that requested /llms.txt. Decide on that evidence, not on a vendor blog post, whether the file is worth maintaining.

Resources

Google Search Central: AI features and your website, read 11 October 2026.
OpenAI: Overview of OpenAI crawlers, read 11 October 2026.
Anthropic: crawler help article, read 11 October 2026.
Perplexity: Perplexity Crawlers, read 11 October 2026.
llmstxt.org: the /llms.txt file, v2, read 11 October 2026.

Does llms.txt Work? What AI Vendors Actually Say About It