llms.txt vs robots.txt: What Each File Does and Which You Need
Robots.txt asks bots to stay out of parts of your website and every AI search engine documents it. llms.txt only suggests what to read. Here is what belongs in each.
- 5 AI engines checked
- A score for every page
- The passage an AI would quote
llms.txt vs robots.txt is a comparison between two plain text files at a website's root that do quite different jobs. Robots.txt is an IETF standard that tells crawlers which paths they may fetch, and the AI search engines document following it; llms.txt is a proposed Markdown reading list for language models that cannot allow or block anything.
- Robots.txt is defined in RFC 9309, published by the IETF in September 2022, which states that its rules "are not a form of access authorization" (RFC 9309, the Robots Exclusion Protocol).
- Google says 0 AI text files are needed to appear in Google Search, and that Google Search ignores llms.txt files (Google Search Central's guide to generative AI search).
- OpenAI says a robots.txt change can take about 24 hours to reach the systems behind ChatGPT search results (OpenAI's crawler documentation).
- Anthropic describes 3 bots, each with its own robots.txt line, and its crawler page does not mention llms.txt (Anthropic's page on how site owners can block its crawlers).
What does robots.txt do that llms.txt cannot?
Robots.txt asks named bots to stay out of named parts of a website, and the major search and AI companies publish how their bots obey it.
The file sits at the top level of a domain, in lower case, and contains groups of rules. Each group names one or more user agents, such as OAI-SearchBot or GPTBot, and lists the paths that bot may or may not fetch. RFC 9309 sets out how crawlers must read those groups: names are matched without regard to case, the most specific matching rule wins, and a crawler should not rely on a cached copy for more than 24 hours.
That shared rulebook is the reason robots.txt carries weight. OpenAI, Anthropic, Perplexity, Google and Microsoft each name their bots and say how robots.txt affects them, so one line can keep a training bot out while leaving a search bot welcome. OpenAI, for example, says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, which makes that single line one of the most important on a business website.
Robots.txt also has limits that owners often miss. The RFC says its rules are not access authorization and are no substitute for real security, so a bot that ignores the file is not stopped by it. Google's own introduction to robots.txt adds that a blocked page can still be indexed if other websites link to it, and that noindex or a password is the way to keep a page out of results.
What does llms.txt do that robots.txt does not?
llms.txt describes a website: it gives a model a summary of the business and a hand picked list of pages, each with a note on what it covers.
Robots.txt says nothing about what any page contains. It can tell a bot that /services/ is open and /admin/ is closed, but not that the services page lists prices for boiler repairs in Nairobi or that the delivery page explains collection hours. The proposal behind llms.txt fills that gap with plain Markdown: a title, a short summary, then lists of links grouped under headings, with a final Optional section a model can skip when space is short.
That makes llms.txt closer to a guided tour than a gatekeeper. It is meant to be read at the moment a model needs information, for instance when a coding assistant looks up a software library's documentation. The proposal also suggests a clean Markdown copy of each page, which saves a model from wading through menus and scripts.
What llms.txt cannot do is change whether a bot may read anything. Leaving a page out does not hide it, and listing a page does not let in a bot that robots.txt or a firewall refuses. It also has no promise of readers among AI search engines: Google says Google Search ignores the file, and the crawler pages from OpenAI, Anthropic and Perplexity describe robots.txt controls without saying their bots read llms.txt to choose sources.
How do llms.txt and robots.txt compare side by side?
The two files share a location and a plain text format, and almost nothing else: one is a standard for access, the other a proposal for reading.
| robots.txt | llms.txt | |
|---|---|---|
| Job | Asks bots which paths they may crawl | Suggests which pages a model should read |
| Status | IETF standard, RFC 9309 | Public proposal, no standards body |
| Format | User-agent groups with Allow and Disallow rules | Markdown title, summary and link lists |
| Can it block a bot? | It asks; well behaved bots comply | No |
| AI search engines that document reading it | Google, OpenAI, Anthropic, Perplexity and Microsoft | None so far; Google says Search ignores it |
| If the file is missing | Under RFC 9309 a crawler may fetch anything | Nothing changes |
| Effect on Google Search | Controls crawling, not indexing | Neither helps nor harms, says Google |
The row about a missing file deserves attention. RFC 9309 says that if robots.txt is unavailable, for example because the server returns a not found error, a crawler may access any resource; if the server fails with an error instead, the crawler must assume everything is disallowed. A broken robots.txt can therefore shut out every bot, while a broken llms.txt affects nobody.
Which file do AI search bots actually read?
AI search bots document reading robots.txt, while none of the five main engines has said its bots read llms.txt to choose what to cite.
Each company splits its bots by job, and robots.txt is where you address each one. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training; Anthropic uses Claude-SearchBot for search and ClaudeBot for training; Google's AI Overviews and AI Mode belong to Google Search, while the Google-Extended token covers Gemini training and grounding. Perplexity says PerplexityBot surfaces and links websites in its results and is not used to crawl content for AI foundation models.
The user fetchers are the exception to the rule. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the request, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says blocking Claude-User stops Claude retrieving your content for a user's question, so its user fetcher does follow the file.
To see how the bots treat your own website, scan it with the AI Search Visibility Checker. It checks robots.txt for each engine's search bot, asks for your homepage as each bot to catch a firewall that turns them away, and lists training bots separately. It reports whether an llms.txt file exists without scoring it, which matches what the engines have said about the file.
What should you put in robots.txt for AI search?
Robots.txt for AI search should let every search bot in, keep private paths closed and, if you wish, refuse training bots by name.
A short file covers most businesses. The example below refuses OpenAI's training bot only, leaves every search bot to the catch all group and points to the sitemap. Swap the domain for your own and add or remove training bots to match your choice.
User-agent: GPTBot # Training only. Delete this group to allow training.
Disallow: /
User-agent: * # Search bots such as OAI-SearchBot and PerplexityBot follow this group.
Disallow: /admin/ # Private paths stay closed to every bot.
Sitemap: https://www.example.com/sitemap.xmlTwo habits keep the file safe. Avoid adding a group for a search bot that only says Allow: /, because under RFC 9309 a bot obeys the group that names it and ignores the catch all group, so it would lose your private path rules. After any change, allow about a day for the engines to notice, and request a page as each bot to confirm nothing in front of the server refuses them.
What should you put in llms.txt, if anything?
If you publish an llms.txt at all, put in it a short, accurate summary of the business and links to the few pages you would want quoted.
Keep it consistent with robots.txt. A page that robots.txt closes to bots should never appear in llms.txt, because the file would then recommend a page no well behaved bot may read. The reverse also holds: if llms.txt lists your pricing page, check that robots.txt and your firewall let the search bots reach it.
Keep it consistent with the pages too. Summaries that promise services you no longer offer, or prices you have changed, give any reader old facts. A hypothetical bakery in Leeds that updates cake sizes on its website but not in llms.txt has created two versions of the truth, and only one of them is maintained.
Most business websites can skip the file without loss, as our guide to llms.txt explains in more detail. A short website that search bots can already reach gains little from a reading list of itself. Software companies with large documentation sets have the strongest reason to publish one, since coding assistants are its main readers.
Can one file replace the other?
No: robots.txt cannot describe your pages, and llms.txt cannot control which bots may read them, so neither replaces the other.
Treat them as two answers to two questions, asked by different readers. Robots.txt answers "may this bot fetch this path?" for crawlers that follow the standard. The llms.txt file answers "which pages should a model read first, and what do they say?" for any tool that chooses to open it.
Neither is a security tool. The RFC notes that listing paths in robots.txt makes them public, so a secret area is better protected with a login than a Disallow line. The same applies to llms.txt: anything in it is visible to anyone who types the address.
Questions owners ask about llms.txt and robots.txt
Do I need both llms.txt and robots.txt?
You need robots.txt, or at least a clear decision to have none, because AI search bots read it. llms.txt is optional. Google says it neither helps nor harms in Google Search, and the other engines have not said they use it.
Can I block AI bots in llms.txt?
No, llms.txt has no rules a bot obeys. Blocking a bot means adding a group for its user agent in robots.txt with Disallow: /. To enforce a block on bots that ignore robots.txt, use your firewall or CDN.
Does Google read llms.txt the way it reads robots.txt?
No, Google says Google Search ignores llms.txt files. Googlebot always obeys robots.txt when crawling automatically, according to Google's crawler documentation. The Google-Extended token in robots.txt, not llms.txt, controls Gemini training and grounding.
Where do the two files go?
Both go at the root of the domain, at /robots.txt and /llms.txt, and the robots.txt name must be lower case. Each subdomain needs its own robots.txt. Anthropic's crawler page, for instance, tells owners to add their rules for each subdomain they want to cover.
Will a robots.txt mistake hurt my AI visibility faster than a missing llms.txt?
Yes, by a wide margin. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, and that robots.txt changes reach its systems in about 24 hours. A missing llms.txt has no documented effect on any engine.
Check both files in ten minutes
Open yourdomain/robots.txt in a browser and read each group, looking for any line that names a search bot or a Disallow: / under the catch all group.
Then confirm that a firewall or CDN is not refusing the bots robots.txt allows, which a scan of your homepage will show. Fix robots.txt first if anything is wrong, because it is the file every engine documents. Allow about a day after any change before checking again.
Then open yourdomain/llms.txt, if you have one. Check that every address works, that no listed page is blocked in robots.txt, and that each summary matches the page today. If the file is out of date and nobody will maintain it, removing it costs you nothing in any engine that has published its position.
Will ChatGPT, Perplexity, Claude, Google AI and Copilot quote your website? Every page scored out of 100, the passage an AI would quote and what to fix first. No sign-up.
Check your website with the AI Search Visibility Checker →