Nobody is reading your llms.txt

Jeremy Howard proposed llms.txt two years ago today, on September 3, 2024. The problem he described was narrow and real: context windows are small, an HTML page is mostly navigation and script, and a model pointed at a documentation site burns most of its budget before it reaches a sentence worth reading. It was a proposal about feeding a model on purpose. Within a year it had been re-read as a proposal about being found.

In June 2025 there were 4,088 sites with an llms.txt file on them. Twelve months later there were 36,120, out of the three million or so that Originality.AI watches. An 8.8x run in a year, for a file no standards body ratified and no platform asked for. The sibling formats grew faster in percentage terms and from almost nothing: llms-full.txt went from 23 sites to 2,463.

Then somebody opened the server logs. Across 137,210 domains that received traffic in May 2026, 97% of the llms.txt files sitting on them were requested zero times that month. Not requested rarely. Requested never, by anything.

Sit with that before the arguing starts. This is the file an entire industry added on the understanding that assistants were reading it. The logs say the assistants were not there. Every number below was checked on September 3, 2026, and every one of them is somebody else’s published data rather than ours, so you can go and disagree with the methodology directly.

The sample is 137,210 domains in Ahrefs Web Analytics with traffic in May 2026. Of those, 28% served a valid llms.txt at the root, which Ahrefs itself flags as an upper bound because its customers skew technical. A wider sweep of close to 300,000 domains by SE Ranking put adoption nearer 10.13%. Pick either figure. Of the files that exist, 97% went a full month without being fetched.

The 3% that did get requests are the interesting part, because of who showed up. Bots made 96% of those requests and humans 4%. The single biggest requester was not an assistant: SEO audit tools, at 21.7%. That is other people’s crawlers checking whether the file was there. General web crawlers took 13.1%, tech profiling tools 11.6%, unidentified bots 14.9%.

AI bots of every description came to 19.5% of requests. Split that out and the slice that matters, retrieval bots, the ones fetching a page to answer a question somebody is waiting on, is 1.1%. GPTBot accounted for 4.51% of requests, ClaudeBot 0.80%, DeepseekBot 0.02%. Those are shares of traffic to the 3% of files that got any traffic at all, which is a small number of a small number.

None of that is the finding that ends the argument. This is: no AI bot requested an llms.txt from a domain that did not have one. Not once, anywhere in the sample. A crawler that goes looking for a file leaves 404s behind it, and there were none to find. Every fetch that landed happened because something handed over the URL first. The file is not discovered. It is passed.

Worth saying who ran that study, since it is not a disinterested party and neither are we. Ahrefs sells prompt tracking, priced out plan by plan against ours in Beseen vs Ahrefs. It is still the best log data anybody has published on this, and it points away from a product either of us could sell you.

The log study answers whether anything reads the file. It does not answer whether having one changes what an assistant says, which is the question people actually mean. SE Ranking went at that side with nearly 300,000 domains and found adoption flat across the board: 9.88% among low-traffic sites, 10.54% in the middle, 8.27% at the top. The big sites are not quietly doing this more than anyone else.

No correlation with citation frequency turned up, by ordinary statistics or by model. The detail worth quoting is what happened when they dropped llms.txt from the XGBoost model: accuracy went up. A variable that makes a model worse by being in it is not a weak positive signal. It is noise dressed as one.

Two more numbers from the same direction. Among the fifty domains assistants cite most, exactly one had an llms.txt file. And in a March 2026 sweep of the Fortune 500, 7.4% had the file while 92.8% had a robots.txt, which is roughly the ratio you would expect between a convention that works and one being discussed.

What none of that proves

Being fair about the shape of this evidence matters, because the shape is the whole argument. These are correlation studies on a convention that is barely eighteen months old. They cannot see a thing that has not happened yet, and if OpenAI shipped support next quarter every figure above would be stale inside a week.

What would settle it is the experiment nobody has published: a set of sites, one prompt set asked before the file goes up and re-asked weekly for a month after, with comparable sites that did not add it as the control. Until that exists, “add llms.txt” is a hypothesis with an impressive adoption curve attached to it. We have been here before, and Ranking is not being named makes the same complaint about articles published to earn citations that nobody ever measured.

Read together, though, the two studies are stronger than either one alone. One says almost nothing fetches the file. The other says sites with it are cited no more often than sites without. A finding with no mechanism behind it would be a curiosity. Here the mechanism and the outcome agree, and they agree on nothing happening.

It is worth going back to the original post, because almost nothing written about this file since has matched what it says. Howard’s pitch was that converting a real HTML page into something a model can use is laborious and error-prone, so a site should offer a curated markdown index instead: the pages that matter, one line of context each, no chrome. His worked example was programming documentation, and the reader he had in mind was a developer assembling context for a prompt.

Read that way, every number in the studies above stops being surprising. A file designed to be pointed at is a file that gets fetched when somebody points at it, which is precisely the pattern the logs show. Nothing about the proposal claimed crawlers would seek it out, and no crawler in the sample did.

The industry supplied that part on its own. Somewhere between the proposal and the tooling, a format for handing a model your docs turned into a ranking factor with a checklist item attached, and 36,000 sites shipped it on that understanding. The file did not fail. The claim made on its behalf failed, and the file has been carrying the reputational damage.

They get filed together and they have almost nothing in common, which matters because one of them can genuinely cost you an answer. robots.txt is a set of instructions about access, honoured by every major crawler that intends to keep its trusted status, and enforced in practice by the CDNs sitting in front of a large slice of the web. Write the wrong line in it and traffic stops.

llms.txt is a curation format. It grants nothing, blocks nothing, and carries no directive of any kind. The worst outcome from writing a bad one is that a model reading it gets a stale list of URLs. The worst outcome from writing a bad robots.txt is that nothing gets read at all, and you will find out through a flat quarter rather than an error message.

That asymmetry is the whole reason for the order of operations later in this post. One of these two files is a nice-to-have that 97% of the internet is not being asked for, and the other is the load-bearing one nobody has re-read since 2023.

Google has been unusually blunt about it. The file is not required for Search and it will not move rankings in either direction. Gary Illyes said in July 2025 that Google does not support llms.txt and has no plans to. John Mueller compared it to the keywords meta tag, which, if you have been doing this long enough to catch the reference, is about as unkind as that comparison gets.

OpenAI and Anthropic both point site owners at robots.txt when the subject is crawler control. llms.txt turns up in their documentation for developers rather than their guidance for publishers, and that placement is the tell. It is a format for handing a model a documentation site on purpose. It was never a discovery channel, and most of the advice written about it in the last year quietly promoted it into one.

There is a real use case and it is worth being precise about, since “it does nothing” is as sloppy as the claim it replaced. Point Cursor, Claude Code or Copilot at a documentation site and llms.txt earns its place immediately, because the human in that loop pastes the URL themselves. That is exactly the behaviour the log study described: the file gets fetched when something hands it over. Stripe, Vercel, Anthropic and Cloudflare all publish one, and they are right to.

So the split is clean enough to act on. If you sell to developers and your docs are most of the sale, llms.txt is a documentation feature with a named reader, and it should be current, accurate and owned by someone. If you are a marketing site selling to anybody else, ship it because it costs an afternoon, then stop thinking about it, and do not let it sit on a roadmap above anything in the next section.

The other honest case is the one that has not arrived: buying agents acting for a person, needing a clean machine-readable view of a catalogue, prices and policies. That is a plausible future and a terrible reason to prioritise the file today, because the version you ship now will not be the version that is read then.

Here is the part the whole llms.txt debate has been standing in front of. If assistants are not naming you, the interesting question was never whether one of them read a text file at your root. It is whether anything could read the page that would have won the answer. Four ways that goes wrong, in rough order of how often we see them.

Your CDN is about to answer on your behalf

On September 15, 2026, Cloudflare’s defaults change. Crawlers get sorted into three kinds: search, which collects and indexes content to answer questions about it later; agent, which acts in real time on a person’s behalf; and training, which takes content to train models. New domains onboarding get training and agent blocked by default on ad-monetised pages, with search left allowed. Existing customers set their own preference in security settings before the date.

The word to notice is agent. That is the live fetch sitting behind an answer somebody is waiting on. A publisher who blocks training and agent together has made a real decision about their business, and good for them. A B2B site that inherits the default because somebody clicked through onboarding at speed has made no decision at all, and will spend a quarter wondering why the rewrites did nothing. Cloudflare sits in front of more than 20% of web domains, so this is not a niche configuration question.

robots.txt says no and nobody remembers writing it

Roughly one site in ten blocks GPTBot in robots.txt, and among the most-trafficked sites it runs closer to a quarter. Almost none of those lines were written this year. They went in during 2023 and 2024, when the argument was about training data and the answer felt obvious at the time.

A disallow written to keep your content out of a training set also keeps you out of the answer, because for several of these crawlers it is one user agent doing both jobs. Reading your own robots.txt as it is written rather than as you remember writing it is a ten-minute job that occasionally explains an entire year of results.

The page needs a browser and the agent did not bring one

Fetch a page the way an assistant does, without JavaScript, and a surprising number of sites return a shell: navigation, a footer, a loading state, and none of the sentence you wanted quoted. The content is real and it renders perfectly for a person. It simply is not in the bytes anything else receives.

This is not the same failure as writing a weak page, and it does not get fixed by writing a better one. Check it before you touch the copy. View source and look for the answer sentence, not for the word count.

200 for Chrome, 403 for everything else

The quietest one. Bot management, a WAF rule or a JavaScript challenge, and a page that loads flawlessly in your browser hands a 403 or an interstitial to anything carrying a non-browser user agent. Nothing looks broken from the inside. Everything is broken from the outside, and the only way to see it is to ask the way a machine would.

None of this needs a tool. It needs log access, curl and about an hour.

• Grep 30 days of access logs for llms.txt. Count the hits, read the user agents, and note how many came from an AI retrieval bot rather than an audit tool. That is your own version of the 97%, and it is the only version that is about your site.
• Fetch your ten most important pages with curl, a bot user agent and no JavaScript. Compare the byte count to what a browser gets, then search the result for the exact sentence you want an assistant to quote. If it is not in there, nothing else on this list matters yet.
• Open robots.txt and read every line, including the ones you inherited. Write down which AI user agents are named and what each disallow actually covers. Do the same for any meta robots or X-Robots-Tag headers on the pages that matter.
• Check your CDN's bot settings before September 15 rather than after, and know which of the three crawler classes you are allowing. Search allowed and agent blocked is a coherent position. Not knowing which one you picked is not.
• Then, and only then, run the prompts. Twenty to thirty of them, worded the way a buyer asks rather than the way a keyword tool phrases things, against ChatGPT, Claude and Perplexity, wording frozen, repeated weekly. One run is an anecdote.

Order matters here more than any individual step. A rewrite aimed at a page nothing can fetch is a week spent on the wrong half of the problem, and the re-check comes back flat with no explanation attached to it. Fix the fetch first, then argue about the copy.

Ten minutes of advice, since the file is cheap and the downside is not zero. A stale llms.txt is worse than none: it is a list of URLs handed to something that will read them literally, so a file that still advertises last year’s pricing page is a wrong answer with your name on it.

• Generate it from whatever already holds the truth. Ours is built from the same constants as the pages, so a price change updates the file without anyone remembering to.
• Keep it short and annotated. It is a map with one line of context per link, not a second sitemap. A dump of every URL you have is the least useful version of this file.
• Check every link in it resolves, on the day you ship it and after any route change. A 404 inside the file is the one failure mode that actively costs you something.
• Only add llms-full.txt if the prose in it exists somewhere machine-readable already. Reconstructing your marketing pages by hand into a text file gives you a second copy to keep in sync, and it will lose.

beseen.so/llms.txt exists, and so does llms-full.txt. Both are generated from the same constants as the rest of the site, so they cannot quietly disagree with a price on a page. They took an afternoon to build and cost nothing to maintain, and there is one reader we can actually name: somebody pasting the URL into an assistant to ask what this product does.

What we will not do is sell it as a lever. We have no before and after showing that file moved a single citation, and neither does anyone else, which is this entire post applied to our own work.

If you want to know which of the four failures above applies to you, the free scan fetches your site the way an agent would and checks whether the pages are usable at all, alongside five prompts on one engine. If it comes back saying the pages are fine and the answers still name somebody else, that is a content problem rather than a plumbing one, and how it works covers what the loop does about it.