llms.txt in Practice: Adoption Data, Evidence, and Setup

llms.txt adoption is not one number, and pretending otherwise is how most coverage of the file goes wrong. Measured across a curated, developer-weighted panel it looks like a settled convention. Measured across the broad traffic-ranked web it looks like a rounding error. Both readings are correct, they were taken from different samples, and the gap between them is the most useful thing the data has to say.

The evidence question is harder and, on the record available at the time of writing, less flattering. The two best-documented studies — one correlating adoption against AI-citation frequency across roughly 300,000 domains, one reading actual server logs across about 900 — both land on a null. No measurable citation lift. No verified requests from a frontier AI lab. Google has said, on the record and in two separate registers, that the file does nothing for Search.

This post reports all of that straight, then gets practical: who has actually shipped the file, who is genuinely requesting it, what the null results do and do not rule out, and a fifteen-minute setup with the five failure modes that make an llms.txt worse than none. Our companion piece on AI crawl economics and pay-per-crawl math covers the same adoption panels from the monetization angle; this one takes the evidence question apart.

llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai and a former Kaggle president, on September 3, 2024. The argument in the original write-up is narrow and unglamorous: context windows cannot hold a whole website, and converting HTML to clean text is, in the spec’s own words, difficult and imprecise. A single markdown file at the site root can give a language model brief background information, guidance, and links to detailed markdown files without any of that overhead.

The format, per the spec at llmstxt.org, is deliberately thin. One H1 heading is the only mandatory element. Everything else is optional: a blockquote summary directly under the H1, free-form detail sections, and H2-delimited lists of links with short descriptions. A special Optional H2 section marks material an agent can skip when its context budget is tight — a genuinely clever affordance that almost nobody uses correctly, for reasons we come back to in Section 08.

The spec also proposes per-page clean-markdown mirrors: the same URL with .md appended, so example.com/docs/api gains example.com/docs/api.md, with index.html.md for extensionless URLs. That half of the proposal is the more demanding one and the less widely implemented, and it is where the file connects to a broader publishing decision we covered in our guide to markdown-first content architecture.

One structural fact frames everything below. As of mid-2026 llms.txt has no W3C, IETF or schema.org recognition. There is no version number, no governance body, and no conformance test. It is an informal community convention that happens to be widely known — which is exactly why adoption data and traffic data are the only honest ways to judge it, and why vendor silence about whether anyone reads it matters so much.

Every published adoption figure for llms.txt is defensible and none of them describes the same web. The table below sets three separate measurements side by side with their sampling frames intact — the fixed 219-host panel, the monthly Tranco top-1,000 tracker, and a 300,000-domain study — because the difference between the frames is the whole story. Averaging these rows would produce a number that describes nothing.

The reconciliation matters more than any single row. When we wrote about Google confirming llms.txt has no SEO value in June, we cited an early scan of that same tracker showing three files across the top 1,000 — about 0.3% — and flagged at the time that later refreshes counted higher. They do: the June 2026 refresh of the same tracker counts 87. Nothing about the earlier number was wrong; it was an early pass at a young measurement, and the honest move is to carry the qualifier rather than quietly restate the headline.

The comparison worth making is the like-for-like one. Both the curated panel and the traffic-ranked tracker can be read on reachable hosts only: 51.8% against 15.8%. That is still a 3.3-fold gap, and it is not noise. It is what happens when you sample the sectors whose readers run coding agents versus sampling the web by traffic. On the raw headline figures the gap is wider still — 51.8% against 8.7%, close to six times — but that comparison quietly mixes a reachable-only rate with an all-hosts rate, which is exactly the kind of blending this section is arguing against.

The sector rows sum cleanly — 218 reachable hosts, 113 with a valid file — which is a decent sign the panel is being reported honestly rather than reconstructed. The top three sectors carry 81 of those 113 files between them. The bottom three carry 5.

Media is the row to sit with: zero of fourteen. The sector with the most acute exposure to AI crawling is the one least interested in publishing a free, machine-readable index of what to crawl. That is not inertia, it is consistency, and it is the clearest evidence in the whole dataset that adoption tracks intent rather than sophistication. Named adopters in the traffic-ranked sample tell the same story from the other side: Cloudflare, Azure, GitHub, Fastly, WordPress.org, Adobe and Samsung — infrastructure and developer platforms, not retail or publishing.

Adoption tells you who published. Server logs tell you who read. The most granular public answer to the second question comes from a server-log study published on April 13, 2026 that monitored roughly 900 domains from September 4, 2025 to April 13, 2026 and counted every request for an llms.txt-family path — /llms.txt, /llms-full.txt, /llms-ctx.txt and a handful of nonstandard variants.

The total across that entire window and that entire estate was 1,227 requests. That is fewer than seven a day across all 900 sites combined. Traffic touched only 107 of the 900 domains, meaning 793 sites — about 88% of the estate — never received a single request for the file across seven months. The canonical /llms.txt path took 1,215 of the 1,227 requests, or 99.0%; every alternative path combined took 12.

The full breakdown is worth printing line by line, because the headline — no AI bots — is easy to dismiss as rhetorical until you see what did show up instead.

Read this carefully, because it is easy to over-claim in both directions. What the log rules out is a publicly identifiable crawl of llms.txt by the major labs across this estate in this window. What it does not rule out is on-demand retrieval — an agent fetching your file mid-task from a user session, with a user agent that does not announce itself as GPTBot or ClaudeBot, or from an IDE-side process that never touches your production logs at all. That distinction is precisely what makes the file plausible for coding agents and implausible for AI-search citation, and it is also why your own logs are the only place to settle it for your site. Our log-file analysis guide covers how to isolate the requests, and the discoverability audit walkthrough shows how a broken redirect or an unreachable root quietly makes the file invisible in the first place.

The second body of evidence attacks the question from the outcome side. If llms.txt helped a site get cited by AI answer engines, sites with the file should be cited more often than comparable sites without it. A study published on November 20, 2025 tested exactly that across roughly 300,000 domains, of which 10.13% had implemented the file.

The finding was a clean null. Correlation tests across major language models showed no measurable link between publishing llms.txt and AI-citation frequency. The researchers then went further and built a gradient-boosted prediction model, on the reasoning that a weak signal might still contribute predictive value even without a strong correlation. It did not: removing the llms.txt feature improved the model’s accuracy. On that evidence the variable was behaving as noise. Their own summary was appropriately careful — llms.txt doesn’t seem to directly impact AI citation frequency, at least not yet.

The “at least not yet” is doing real work and deserves to be taken seriously rather than treated as hedging. A null result on an observational dataset is not proof of no effect. It is evidence that any effect is smaller than the noise floor of a 300,000-domain sample, which for a practitioner is the same thing as no effect for planning purposes — but it says nothing about what happens if a major provider decides next quarter to start reading the file. That is the honest boundary of the claim.

Two independent methodologies — outcome correlation and raw request logs — reaching compatible nulls is a stronger result than either alone. They fail differently: a correlation study can miss an effect hidden by confounders, a log study can miss traffic that never reaches the origin. That they agree is the reason to plan around the null rather than wait for better news. The same discipline applies to structured data, where we walked through what schema markup can and cannot be shown to do once you insist on evidence rather than convention.

Google’s position gets reported as a contradiction. It is not. The company has said two things through two different organisations with two different jobs, and the tension between them is the most useful signal available about what the file is actually for.

Statement one, from Search. Google’s documentation update in June 2026 states that llms.txt has no effect — positive or negative — on Search rankings or AI Overviews; Search simply ignores the file. We covered that update and the reasoning behind it in detail when Google confirmed llms.txt does nothing for rankings. Separately, John Mueller of Google Search Relations explained the structural reason on the Search Off the Record podcast, in an episode reported on June 15, 2026: a self-reported manifest cannot function as a differentiator between sites, because every site would claim to be the best one.

Mueller allowed one narrow exception in the same conversation, and it is the sentence practitioners should build on rather than the headline: if someone is already on your website, some kind of automated system may be helpful. That is a precise carve-out. The file is not a way to be found. It may be a way to be navigated once an agent has already arrived. Gary Illyes, also of Google Search, put the SEO value more bluntly still, calling the format at best neutral and comparing it to the keywords meta tag — an analogy chosen with some care, since that tag is the canonical example of a self-reported signal search engines stopped trusting precisely because it was self-reported.

Statement two, from Chrome. On May 5, 2026 Google added an llms.txt audit to Lighthouse, filed under a new agentic browsing audits category alongside WebMCP, accessibility and layout-stability checks. The audit’s behaviour is narrow and revealing: it flags a page only when the server returns an error while fetching /llms.txt. A plain 404 marks the audit Not Applicable, because the file remains entirely optional. Chrome’s own documentation states that without the file, agents may spend more time crawling the site to understand its high-level structure and primary content.

The forward read is the interesting part. Google has effectively placed two bets in the same drawer — a passive text manifest that anyone can publish in fifteen minutes, and a structured, callable agent affordance with browser support and an origin trial. Over the next several quarters those two will diverge on adoption curves that are worth watching side by side, because they answer the same question with very different amounts of commitment. If agent-facing site quality becomes a real discipline, the structured affordance is the one with a governance story and a browser behind it. The manifest is the one you can ship this afternoon, which is not nothing, but is a different kind of bet.

Everything above is a set of negative findings, and negative findings are where most llms.txt coverage either stops or starts inventing. The positive case is real but much narrower than the pitch, and it should be stated with the sourcing it actually has.

Two caveats on that first card, because they matter more than the card does. The coding-agent use case rests on secondary characterisation rather than a benchmarked mechanism — we could not locate vendor documentation stating precisely when a given assistant fetches the file or what it does with the contents. And the vendor silence cuts both ways: at the time of writing we found no primary, on-record statement from OpenAI, Anthropic or Perplexity either confirming or denying that their crawlers fetch and parse /llms.txt. The absence of a denial is not evidence of reading, and the absence of a confirmation is not evidence of ignoring. It is simply an open question that nobody with the logs has answered publicly.

There is also a structural reason to expect provider caution that has nothing to do with llms.txt specifically. Published academic work on adversarial optimisation against language models has shown that crafted content can meaningfully shift how likely a model is to recommend a source. Any system that ranks or selects sources has an obvious reason to discount inputs the publisher wrote about themselves. That is the same logic Mueller articulated, arrived at from the security side rather than the search side, and it is why we would not expect a self-reported manifest to become a retrieval signal even if adoption doubled.

If you are a developer-facing site — docs, an API, an SDK, a developer-adjacent SaaS — the file is worth building, on the strength of the coding-agent audience and nothing else. It is genuinely a fifteen-minute job if you do not gold-plate it, and the three blocks below are the whole scope.

The cap is the only part of this that requires judgement, and it is worth thinking about as a context-budget problem rather than a completeness problem. The spec’s Optional H2 section exists exactly for this: material a reader can skip when it is short on budget. Most implementations ignore it and dump everything into a flat list, which inverts the point of the file. A curated forty-link map with a summary line each is more useful to a context-limited reader than a two-thousand-line index, and it is also cheaper for you to keep accurate.

One decision to make deliberately rather than by default: whether the file coexists with your crawl-control policy or quietly undermines it. Publishing a curated index of your best content is a different posture from restricting who may fetch it, and the two need to agree. Our AI crawler access-control decision matrix works through the robots.txt side of that call bot by bot.

The file is cheap to ship and cheap to get wrong. These are the five failure modes worth designing against, in rough order of how often we see them.

The through-line across all five is that llms.txt is a maintenance commitment disguised as a one-off file. That is the honest reason to skip it if you are not developer-facing: the build is fifteen minutes, the upkeep is forever, and the measured return outside the coding-agent audience is a null result from two independent methods. If you need help drawing that line for your own stack — which surfaces are worth optimising for machine readers and which are not — that is the substance of our agentic SEO engagements, and the publishing side of it lives in our content engine work.