What is llms.txt, and should your website have one?
A plain-markdown map of your site for AI systems — cheap to ship, zero downside, and one honest caveat about who reads it today.
llms.txt is a plain-text markdown file at the root of your website — yoursite.com/llms.txt — that gives AI systems a concise, curated map of who you are and where your canonical pages live. Think of it as the opposite of robots.txt: robots.txt tells crawlers what to ignore; llms.txt tells language models what matters. It costs an hour to ship. I ship it on my own sites.
Where it came from
The proposal originated with Jeremy Howard of Answer.AI in late 2024: give large language models a file designed for them, the way sitemaps were designed for search crawlers. HTML pages are a terrible format for a model with a limited context window — navigation, scripts, cookie banners, and boilerplate bury the substance. A markdown file with a short description and a curated link list is close to the ideal input.
The convention has two parts. llms.txt is the concise map: a title, a one-paragraph summary, and grouped links to your canonical pages with a line of context each. llms-full.txt is the expanded version — the full content inlined, for systems that want everything in one fetch.
What goes in mine
On carlfarrington.com, the file answers the questions I want machines to get right: who I am, in one tight paragraph; what I run, with links — the news network, the platform, the agency, the tools; and where the durable pages live — the about page, the timeline, these dispatches. Every line exists to disambiguate. Language models don't struggle to find text about you; they struggle to decide which text is authoritative. This file is you telling them.
Same pattern for a business: what the company does in plain words, the canonical product and pricing pages, the docs, the contact path. Leave out the marketing fluff — a model quoting your llms.txt verbatim should sound accurate, not promotional.
The honest caveat
Here's what the people selling "AI SEO packages" won't lead with: as of today, no major AI vendor has committed to reading llms.txt. OpenAI, Anthropic, Google, and Perplexity crawl your regular pages; whether their pipelines consult llms.txt is somewhere between "unconfirmed" and "sometimes." Anyone promising it's a ranking factor is ahead of the evidence.
So why do I ship it anyway? Three reasons.
The cost is nearly zero. It's a static text file. There is no downside scenario.
The bet is asymmetric. Adoption is a chicken-and-egg problem — tools and crawlers add support as more sites publish the file. If it becomes standard, early sites are already legible. If it dies, you spent an hour.
Writing it is valuable even if no machine ever reads it. Forcing yourself to state, in one page of plain markdown, who you are and which pages are canonical is an entity-clarity audit. Most sites can't do it — which tells you why AI systems describe them incorrectly.
Should you add one?
If you run a site where being described accurately by AI systems matters — a business, a professional identity, a product — yes. Do it after the fundamentals, not instead of them: your real pages still need clean HTML, real dates, schema markup, and a quotable opening paragraph, because those are what today's crawlers demonstrably read. llms.txt is the cheap bonus layer on top, not the strategy.
One hour, one text file, no downside. In a field full of expensive maybes, that's about the best trade on the board.