Llms.txt Explained: The Proposal, Practical Uses and Limits

Begin with the proposal's actual status

Llms.txt is a proposal authored by Jeremy Howard, with a v2 revision dated 10 August 2026. The current specification permits a file at the site root or within a path, describing content under that path. Its purpose is to offer agents a concise overview and useful links. Calling it a proposal matters: publication of a convention does not establish that every search engine, model or browser agent consumes it. A site can follow the format accurately and still have no evidence that a particular external service uses the file in answering questions.

Separate three questions that often get mixed together. Does the file exist and follow the proposed format? Does a specific tool actually read it? Does reading it improve a measured outcome? Evidence for the first does not establish the second or third. A file can be useful to an owned documentation assistant even if it has no effect on public search. Conversely, an automatic generator may create a valid file that nobody uses. Define the intended consumer and task before investing in a large implementation or presenting file publication as a completed visibility strategy.

Understand what the current file contains

The v2 proposal uses Markdown, beginning with a project or site H1. It describes optional summary material and sections containing links to further information. It also proposes Markdown alternatives and discovery link relations, including a relation pointing to the relevant llms.txt file. These are concise format facts from the current specification, not a guarantee of consumer adoption. If you implement the convention, verify the current specification rather than copying an old tutorial that assumes only one root-level file or treats every historical detail as a permanent requirement.

For an illustrative software documentation site, a useful file might identify the product, distinguish current documentation from archived versions and link to installation, API reference and migration guides. Each description should explain what the destination offers. A long unordered dump of every URL may add little beyond existing navigation. Keep the file focused on helping its identified reader choose the right resource. The anchor text guide applies naturally here: descriptive link labels reduce the effort needed to decide which destination answers the task, whether the reader is a person or a software agent.

Keep it separate from crawling and indexing controls

Llms.txt is not a substitute for robots rules, authentication or a search inclusion directive. Do not place confidential material in it under the assumption that only approved models can see it. Treat a publicly served file as public information. If a resource should be private, protect the resource through the appropriate access mechanism. The AI crawler controls guide explains how access policies differ across consumers. A helpful index can describe allowed public resources, but descriptive guidance does not enforce who can fetch them or what an external system will subsequently do with the content.

Likewise, a curated guide and an XML sitemap have different operational purposes. The file may highlight a small set of useful documents, while a sitemap supports discovery of the intended public URL set for compatible search systems. Maintain normal crawlable pages, internal navigation and coherent canonical destinations. The XML sitemap guide covers that established workflow. If the site has broken links or inaccessible primary content, publishing a supplementary text file does not repair those failures. Prioritise the information path that readers and verified consumers actually need rather than letting an optional artefact distract from a confirmed access problem.

Check consumer-specific claims before promising value

Google's current generative AI search guidance states that Google Search does not use llms.txt or similar special files for visibility or ranking. That statement is specific to Google Search. It does not establish that no other tool can use the convention. Keep both boundaries intact: do not sell the file as a Google ranking requirement, and do not dismiss a documented use by a different consumer simply because Google ignores it. The relevant decision depends on the actual application, its instructions and the task the file is supposed to support.

If a vendor says its agent reads llms.txt, ask for the documented behaviour or test it in an environment you control. Does it discover the file automatically, require an explicit URL or merely accept the text when pasted into context? These are different capabilities. A successful manual fetch does not prove automatic use during ordinary answers. Log or inspect the tool's actual requests where you are authorised to do so. The multi-platform testing guide helps avoid collapsing different product behaviours into one claim about AI systems as a whole. Report the consumer and method with any result.

Build one maintained representation of the truth

Avoid manually maintaining a second set of important product facts if the main site already has a reliable source. Generate stable link lists from the same content records where feasible, while giving an editor control over descriptions and prioritisation. A Markdown alternative should agree with its public HTML counterpart on conditions, dates and instructions. If the two diverge, an agent may receive obsolete information even while human readers see the corrected page. Content governance should assign responsibility for this additional representation and include it in the correction path for shared facts.

Test the file as a resource after deployment. Confirm that it returns the intended content, that linked destinations exist and that path-scoped coverage matches the site structure. Do not accept a success response containing the site's generic application shell as evidence that the file was served correctly. Check any proposed discovery headers or links against the current implementation. These are ordinary technical release checks, with the same need to distinguish a generated asset from its live production response. Keep the implementation small enough that ongoing verification does not exceed the plausible benefit to the identified consumer.

Make adoption a bounded experiment

Write a narrow hypothesis, such as helping an owned documentation agent choose the current migration guide when both current and archived guides exist. Define the questions, expected destinations and comparison method before adding the file. Observe whether the agent reads it and whether navigation improves. If no consumer uses it, that is a useful result about this experiment, not proof of permanent universal irrelevance. If a result improves, check whether the improvement came from clearer underlying links or other simultaneous changes. The AEO experiments guide explains how to keep such observations from turning into unsupported causal claims.

Include maintenance effort in the decision. A small generated index serving a known agent can be worthwhile; a manually curated parallel documentation system may create avoidable drift. Revisit the implementation when the consumer changes or the proposal evolves. Keep the normal website accurate and navigable regardless of the experiment's outcome. Llms.txt is best assessed as an optional information interface with specific consumers and upkeep costs. That framing allows a practical yes or no decision without inflating a proposed convention into a universal requirement, or pretending that publishing one text file controls how every future AI answer represents the site.

Sources and further reading

• Jeremy Howard: llms.txt v2 proposalThe August 2026 proposal supports path-scoped files, Markdown links and discovery link relations; it remains a proposal.
• Google: generative AI search guidanceGoogle continues to recommend foundational SEO and does not use special AI text files for Search.