Does Google use llms.txt for rankings or AI Overviews?

Google's page on AI features and your website, consulted on 20 October 2026, says you don't need to create new machine readable files, AI text files or markup to appear in AI Overviews or AI Mode. Google does not describe llms.txt as an input to rankings or to those features. A Chrome Lighthouse audit checks whether the file exists, but that is a diagnostic tool, not a Search signal.

Is llms.txt the same as robots.txt?

No. robots.txt tells compliant crawlers which paths they may request, so it controls access. llms.txt is a Markdown file that points language models to a curated set of pages and gives them context. It grants and denies nothing: if robots.txt or a CDN rule blocks an AI crawler from a page, listing that page in llms.txt does not lift the block.

Do I need llms-full.txt as well as llms.txt?

Not necessarily. llms-full.txt, a single document holding the full text of the listed pages, is a convention used by documentation platforms rather than part of the llms.txt specification. It helps coding agents that want a whole documentation set in a single request. For a marketing site, a well-curated llms.txt that links to clean pages is enough, and it avoids another copy of your content to keep in sync.

Should llms.txt be indexed by Google?

Either choice is defensible, but noindex is the safer default for most sites. A .txt file cannot carry a robots meta tag, so use an X-Robots-Tag noindex HTTP header, which Google documents for non-HTML files. Noindex keeps a bare list of links from ranking in place of your real pages. Leaving it indexable keeps the file available to retrieval systems that rely on a search index.

How can I tell if AI crawlers read my llms.txt file?

Check your server logs. Filter requests to /llms.txt by user agent, using the names each vendor publishes; OpenAI's crawler overview lists GPTBot, OAI-SearchBot and ChatGPT-User. Record several weeks before publishing and the same length of time after, then compare. A request shows the file was fetched, not that it influenced an answer, so pair the logs with citation checks on a fixed set of prompts.

What an llms.txt File Is and Whether It Helps AI Visibility

An llms.txt file is a Markdown file at a site's root that points language models to a curated set of pages when they answer questions, not when they train. Google and OpenAI do not document reading it, and no published study links it to more citations, so treat it as a low-cost test, not a visibility lever.

What an llms.txt file is and what the specification requires

The proposal comes from Jeremy Howard of Answer.AI and is published at llmstxt.org. It targets inference time: when an assistant or coding agent answers a question about a site, the file tells it which pages matter and where clean versions live. It says nothing about training data or permissions.

Markdown is the format because a model reads text, not layout. An HTML page wraps its content in navigation, scripts, styling and repeated templates, and all of that markup consumes tokens in a context window of fixed size. Markdown keeps the headings, lists and links that carry meaning and drops the rest.

The file is a curated selection, not an inventory. A sitemap lists every URL you want crawled; llms.txt lists the pages a model needs to understand what you offer.

The required and optional parts of the file

The specification, consulted on 20 October 2026, is explicit about what is mandatory: "An H1 with the name of the project or site. This is the only required section." The rest is recommended structure, in this order:

A blockquote with a short summary of the site, written so a model can read it alone.
Optional free-text paragraphs with context the summary cannot hold, such as how to read the file.
Sections under ## headings, each holding a Markdown list of links, with a colon and a short description after each link.
A section titled Optional, whose links an agent can skip when it needs a shorter context.

A minimal file in that format:

# Example Analytics

> Example Analytics measures product usage for SaaS teams.

## Docs

- [Quick start](https://example.com/docs/quick-start.md): install the tracker and send an event
- [API reference](https://example.com/docs/api.md): endpoints, authentication and rate limits

## Optional

- [Changelog](https://example.com/changelog.md): release notes

Anthropic's developer documentation file, consulted on 20 October 2026, opens with the heading "Anthropic Developer Documentation" and the blockquote "This file provides an overview of the Anthropic API documentation and developer resources."

llms.txt, llms-full.txt and Markdown copies of pages

Companion formats around llms.txt do not have equal standing. The specification proposes that useful pages also exist as clean Markdown at the same URL with .md appended, which is why the example links to .md files. llms-full.txt, a single document holding the full text of the listed pages, is a convention adopted by documentation platforms and tools, not part of the specification. It suits agents that want everything in a single request, but it grows large and adds another copy of your content to keep in sync.

How llms.txt differs from robots.txt and sitemap.xml

These root files are often grouped together, but they answer different questions.

Criterion	llms.txt	robots.txt	sitemap.xml
Purpose	Point models to curated pages and context	Tell crawlers which paths they may request	List URLs you want crawled and indexed
Format	Markdown	Plain-text directives	XML
Who reads it	Assistants and agents that choose to fetch it	Compliant crawlers, including AI crawlers	Search engine crawlers
Controls access	No	Yes, for crawlers that respect it	No
Documented by a major engine	Not in Google's or OpenAI's documentation consulted on 20 October 2026	Yes	Yes

llms.txt is a navigation and context aid. It grants nothing and denies nothing. A robots.txt disallow for an AI user agent still applies to the pages it covers, including pages listed in llms.txt, and the same holds for CDN-level blocks such as Cloudflare's AI crawler controls. Decide first which AI crawlers to allow or block in robots.txt, then list only pages those crawlers can fetch.

llms.txt is also an ordinary URL that Googlebot can index and show in results. Google's robots meta tag documentation, consulted on 20 October 2026, states: "You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible." A .txt file cannot carry a meta tag, so send X-Robots-Tag: noindex with /llms.txt to keep it out of the index. Noindex stops a bare list of links ranking in place of your pages; leaving it indexable keeps it findable by retrieval systems built on a search index. For most sites, noindex is the safer default.

What the evidence says about llms.txt and AI citations

What Google and AI vendors have stated

Google's page on AI features and your website, consulted on 20 October 2026, says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The features are AI Overviews and AI Mode. The statement does not say whether Google reads llms.txt; it says such files are not needed.

OpenAI's crawler overview, consulted the same day, lists GPTBot, OAI-SearchBot and ChatGPT-User with their purposes and does not mention llms.txt. This article relies on no statement from Anthropic or Perplexity on whether their crawlers read the file. Anthropic publishes its own llms.txt for its documentation, which is a publishing choice, not a crawling commitment.

A Google tool does check for the file. Lighthouse documentation, consulted on 20 October 2026, lists in its agentic browsing scoring: "llms.txt: Checks for the presence of a machine-readable summary at the domain root." A diagnostic check for a file shows nothing about whether Google Search uses it.

What large-sample studies measured

SE Ranking's study, consulted on 20 October 2026, reports: "Out of the nearly 300,000 domains we analyzed, only 10.13% had an LLMs.txt file in place." Its title gives the authors' conclusion on whether the file helps: "Why It Doesn't Work."

The finding has limits. A correlation study across many domains can show that sites with the file are not cited more often on average; it cannot prove the file has no effect for any site, query type or engine. A single-site log experiment describes a single site over a single period and cannot be generalised. No published evidence we can source shows a measured lift in AI citations from publishing the file, so treat it as a low-cost experiment.

Why a file can be indexed but rarely fetched by AI crawlers

These are different mechanisms, and both can be true at once. When Googlebot indexes /llms.txt as a normal URL, the file can surface in search results, and in retrieval-based answers that draw on a search index, like any other page. An AI engine fetching llms.txt as a site map and following its links is a separate behaviour, and none of the vendor documentation cited here describes it. Seeing the file in an answer does not show that an engine used it as a map.

Which sites should publish llms.txt and which can skip it

Developer documentation and API references are the clear fit: coding assistants and agents read docs at inference time, often with limited context, which is the situation the file was designed for. Elsewhere, the verdict depends on whether a model needs help to find the right pages.

Site type	Verdict	Reason
Developer documentation, API reference	Publish	Agents read docs mid-task, and a curated index with Markdown copies saves context
SaaS marketing site	Test	Product, pricing and comparison pages suit a summary, but the citation effect is unmeasured
E-commerce catalogue	Usually skip	A large, changing catalogue cannot be curated by hand; test with category and policy pages only
Publisher	Usually skip	Articles change too often for a curated list to stay useful
Local business	Skip	A small site with clear HTML and accurate business data gains little

The specification places the file at the root path and, optionally, in a subpath; it sets no rule for languages or brands. In practice, publish a file per host or subdomain, since crawlers treat each host as a separate site. On a multilingual site sharing a host, add a section per language to the root file, or place files under language paths such as /fr/llms.txt and link them from the root file. Give each brand domain in a group its own file.

Weigh the cost against signals engines do document. Crawlable HTML, structured data that matches visible content and an accurate sitemap are documented by search engines; llms.txt adds a file that must change whenever its pages change. Work through the twelve GEO audit checks to run before editing content first. If you then publish, keep the file current: links to redirected, removed or stale URLs send models to the wrong place.

Where to start this week: build, validate and measure your llms.txt

Build the file from your most useful pages

List the questions a model is likely to face about your site: what you offer, for whom, at what price, how to start and how to integrate.
For each question, pick the page that answers it best. Leave out tag pages, paginated archives and anything behind a login.
Write the H1, the summary and grouped link lists as in the example above, with a plain description per link and secondary pages under Optional.
Upload the file to the site root by hand, or with a CMS plugin or generator. Yoast's documentation, consulted on 20 October 2026, says enabling its feature "will create an llms.txt file in the root directory of your website" and that the file "will be updated weekly by a scheduled action". Compare its page selection with your own list.
Tie the file to your release process so it is reviewed whenever pages are added, renamed, redirected or removed.

Validate status code, content type and links

Before treating the file as live, check that:

/llms.txt returns a success status directly, not a redirect or an error.
The Content-Type header is text/plain or text/markdown, with the character encoding declared.
The file has a single top-level heading with the summary directly under it.
Every listed link returns a success status without a redirect and points to a page your robots.txt and CDN rules let AI crawlers fetch.
You have decided whether to send X-Robots-Tag: noindex, and the header matches that decision.

Measure AI crawler requests and referrals before and after

Record a baseline of several weeks before publishing so normal variation is visible, then compare a window of the same length afterwards. Track these signals:

Server-log requests to /llms.txt and the listed pages, filtered by AI user agent. OpenAI's crawler overview lists GPTBot, OAI-SearchBot and ChatGPT-User, describing the last as "used for certain user actions in ChatGPT and Custom GPTs." Add the agent names other vendors publish.
AI referral sessions; to separate them from other traffic, set up an AI referral channel in Google Analytics.
Citations on a fixed prompt set, asked in the same engines with the same wording in both windows.

A change between windows is not proof of cause: model updates, seasonality and your own content changes move the same metrics. Hypothetical example, not a Namedrop result: a documentation site sees ChatGPT-User requests to /llms.txt after publishing while citations on its prompt set stay flat. That shows the file is fetched, not that it earns citations. This article reports no Namedrop observation on llms.txt.

Before and after you publish an llms.txt file, run a free Namedrop scan to record which engines cite your brand today.