What Is llms.txt and Does It Work? Who Reads It, How to Measure It and How to Manage AI Bots with robots.txt

What is llms.txt? (Short answer)

llms.txt is a plain-text file at your site's root (for example yoursite.com/llms.txt) that tells large language models (LLMs) and AI tools 'these are the most important pages on this site'. It is written in Markdown: an H1 title (site or project name), a one-sentence summary and annotated link lists under H2 sections.

Jeremy Howard of Answer.AI proposed the idea on llmstxt.org in September 2024. The goal was to give AI a clean, short content map instead of HTML pages full of ads, menus and JavaScript. One important detail: llms.txt is not a W3C or IETF standard, and no AI company has formally committed to reading it.

The format, an example file and which pages to include are covered in detail on the llms.txt Generator page; this article focuses on whether it works, who reads it and how to measure it.

Does llms.txt work? What does the 2026 data say?

The two largest studies so far reach the same conclusion: llms.txt does not measurably increase visibility in AI search, and only a tiny share of AI search bots even request it.

Ahrefs (June 2026) analysed server logs for 137,210 domains that received traffic in May 2026. About 28% published an llms.txt file (Ahrefs calls this an upper bound because its users skew technical). 97% of those files received no requests during the month; of roughly 22K requests to all files, 96% came from bots — and 77% of those bots were not AI tools but SEO audit tools, general crawlers and tech-profiling bots.

SE Ranking (November 2025) compared llms.txt presence with how often about 300K domains are cited in LLM answers. 10.13% of domains had the file and there was no relationship between the two; removing the llms.txt variable actually improved the model's accuracy.

• In the Ahrefs data, all AI-related categories combined (agents, training crawlers, assistants, retrieval bots) made up 19.5% of requests — which the authors call 'the most generous possible reading', because fetching a file is not the same as reading it
• Among AI bots, GPTBot requested the file most (4.51%); ClaudeBot 0.8% and OAI-SearchBot, ChatGPT search's crawler, 0.74%. PerplexityBot requested llms.txt less often than Slack's link-preview bot
• The notable exception: Claude Code, a coding agent, made more requests than every AI retrieval, assistant and training bot except GPTBot — today's real readers of llms.txt are developer tools
• 98% of requests for missing llms.txt files (404s) came from humans and none from AI bots: if the file isn't there, AI bots don't even look for it

What does Google say about llms.txt?

Google Search's position is clear: its AI search optimisation guide, published in May 2026, says you don't need llms.txt, AI-specific text files, special schema or content chunking to appear in AI Overviews and AI Mode. In June 2026 John Mueller called llms.txt 'purely speculative for now' on Reddit and added that, for most sites, the bigger issue will be blocking agents. He had earlier compared the file to the keywords meta tag.

The confusing part is on the Chrome side: Lighthouse 13.3 (May 2026) added an llms.txt audit to its experimental 'Agentic Browsing' category and describes the file as an 'emerging convention'. It is an experimental check aimed at future browser agents, not a ranking or AI Overviews signal in Google Search. Seeing that warning in Lighthouse does not mean your site is falling behind in search.

Should you add llms.txt? A decision checklist

llms.txt costs little and its return is uncertain today. So the decision should not be 'everyone add it' or 'completely useless' — it depends on who your site is for.

Don't ignore the risks either: an unmaintained file points to removed pages, and Ahrefs recommends treating the file like code — reviewing changes — because it can contain text that AI tools might read as instructions (prompt injection).

• Add it (priority): API or developer documentation, SaaS product docs, open-source projects — Claude Code and similar agents are real visitors of these sites
• Optional (a 30-minute job, low expectations): content-led B2B sites, consulting and service sites, especially if you have a tidy list of tools and guides
• Not a priority: e-commerce and local business sites — product feeds (Merchant Center), Google Business Profile, indexing and page speed matter far more
• Don't: treat llms.txt as access control instead of robots.txt, build separate Markdown pages for bots while neglecting your real pages, or stuff every URL into the file

What's the difference between llms.txt, robots.txt and sitemap.xml?

All three live at the root but do different jobs and don't replace each other. Granting or restricting AI bot access is done with robots.txt, not llms.txt.

• robots.txt: tells each bot which paths it may crawl (access rules). Most search and AI bots follow it
• sitemap.xml: gives search engines the full list of URLs to index and their update dates (discovery)
• llms.txt: offers a short, annotated 'most important pages' summary for people and AI (context). No bot is obliged to follow it

How do you manage AI bots with robots.txt?

AI companies use different bots for different jobs, and each is managed separately in robots.txt. The key split: bots that crawl for model training, bots that crawl for AI search results, and bots that open a page on demand when a user asks a question. According to OpenAI's documentation these settings are independent — you can block training and still appear in search.

Pick your policy first, then add the matching blocks to robots.txt. Changes take time to apply (OpenAI says about 24 hours for search). User-initiated fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt; if content truly must stay private, the only fix is to take it off the public web.

• Training bots: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini Apps and Vertex AI; Google states this token does not affect inclusion or ranking in Google Search)
• AI search bots: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude search), PerplexityBot (Perplexity search). Blocking them reduces your visibility in that AI search
• User-initiated bots: ChatGPT-User, Claude-User, Perplexity-User — they open the page when a user asks; Anthropic notes that blocking Claude-User may reduce visibility for user-directed web search
• Policy 1 — Open to all (recommended for most business sites): no extra rules; 'User-agent: *' with 'Allow: /' is enough
• Policy 2 — Visible in AI search, closed to training: add 'Disallow: /' under each of 'User-agent: GPTBot', 'User-agent: ClaudeBot' and 'User-agent: Google-Extended'; leave OAI-SearchBot, Claude-SearchBot and PerplexityBot alone
• Policy 3 — Closed to all AI bots: give 'Disallow: /' to all training and search bots above, knowing you also give up appearing as a source in AI search

Is your llms.txt being read? How to measure it in server logs

This is Ahrefs' own advice: check your logs before deciding. The steps below show who requested your file over a month. Note: static hosts such as GitHub Pages don't provide access logs; you'll need your CDN's bot reports or hosting that keeps logs.

Read the result carefully: a bot requesting llms.txt does not mean it uses the content or will cite you. The meaningful question is whether AI bots that fetch llms.txt then visit the pages listed in it. In the same logs, compare OAI-SearchBot, Claude-SearchBot and PerplexityBot requests to those pages with the month before you added the file.

• 1. Download the last 30 days of access logs (Nginx/Apache access.log or raw logs from your hosting panel)
• 2. Count llms.txt requests by user agent: grep "llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
• 3. Filter out fake bots: user agents are easy to spoof. OpenAI and Google publish their bots' IP ranges; verify IPs against those lists or with a reverse DNS lookup
• 4. Check status codes: 200 means the file was served, 404 means it doesn't exist; server errors (5xx) also trigger a warning in the Lighthouse audit
• 5. Compare: count AI bots requesting robots.txt and your actual pages in the same period; if llms.txt requests are a small fraction of that, the file is not a reading channel for you right now

What actually helps you appear in AI search?

Google's AI search guide says optimising for generative AI features is part of SEO, not a separate discipline. Being indexed and eligible for snippets is a prerequisite for being cited in AI Overviews and AI Mode. Across other AI search products the common denominator is the same: the bot must be able to reach the page, and the page must contain original information worth citing.

The order we follow on this site: first, publish pages as full HTML generated at build time and get them indexed; then put a direct answer and our own worked examples in the first section of every article; low-cost extras like llms.txt come last.

• Access: don't block AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) or Googlebot in robots.txt; check that firewall or bot-protection settings don't cut them off
• Indexing: important pages must be indexed in Google; regularly clear the 'Discovered – currently not indexed' list in Search Console
• Readable HTML: content that exists in the HTML without running JavaScript (server-side or static generation) matters for many AI bots
• Original content: what Google calls 'non-commodity' content — your own data, worked examples and experience. Repeating the definition everyone writes doesn't earn citations
• Answer-first structure: question-style headings with a clear answer in the first sentence of each section is the easiest format to cite, for readers and AI summaries alike

How do you create llms.txt? (The short version)

If you decide to add it, the job takes under half an hour: put your site name in the H1 and a one-sentence summary below it, list your 20–50 most important pages as annotated links under H2 sections such as 'Tools', 'Guides' and 'Services', and upload the file to the root. Then open yoursite.com/llms.txt in a browser and check it displays as plain text.

The llms.txt Generator does these steps with a form, lets you start from an example file and checks the H1, summary, link format, relative or http links and duplicate URLs. If you use a static site generator, generating the file from your page inventory at build time removes the stale-link risk; on this site llms.txt is regenerated on every build.