GuidesIndexed

What is llms.txt?

Plenty of sites have one now. Almost nothing reads them. Here’s what the file is for, who does read it and the three things to fix before you bother.

Josh Pigford, founder of FirstIndex5 min read
The short answer

llms.txt is a plain Markdown file at yoursite.com/llms.txt that lists your most important pages for AI tools to read. Jeremy Howard proposed it in September 2024. Google says Search doesn’t use it, and no AI answer engine has said it does. It’s cheap to add, but it won’t get you named in AI answers on its own.

What goes in an llms.txt file?

Very little, by design. The proposal at llmstxt.org asks for one thing: a title line (an H1) with your site’s name. Everything else is optional: a one-line summary, a few sentences of context, then sections with a heading each, holding a list of links with a short note after a colon. A section called “Optional” holds links a tool can skip when it’s short on room. The proposal also suggests a plain Markdown copy of each page at the same address with .md on the end.

The parts of an llms.txt fileFour lines from FirstIndex’s llms.txt, each labeled. “# FirstIndex” is the H1 with the site’s name, the only required part. “> AI visibility and SEO for…” is an optional one-line summary. “## Guides” starts an optional section of links. “- [SEO for startups…](…): The order…” is one link with a note after the colon.Required: your site’s name, as an H1# FirstIndexOptional: a one-line summary> AI visibility and SEO for…Optional: a section of links## GuidesEach link, with a note after a colon- [SEO for startups…](…): The…
Lines from our own llms.txt, cut short to fit. Format from the llms.txt proposal.

To make one, write it in any text editor, save it as llms.txt and put it at the root of your site, so it loads at yoursite.com/llms.txt as plain text. Link only to pages that load. On WordPress, Yoast SEO can generate one for you.

Does llms.txt help you show up in AI answers?

Not that anyone can measure. Google’s guide to AI features files llms.txt under things you can ignore: “Google Search itself doesn’t use them.” Google’s John Mueller was blunter. He called it “comparable to the keywords meta tag – this is what a site-owner claims their site is about.”

The big studies agree. Ahrefs looked at 137,210 sites in May 2026, and of the roughly 38,000 with a valid llms.txt, 97% got no traffic to the file at all. SE Ranking checked nearly 300,000 domains and found no correlation between having one and getting cited by AI.

We came at it from the other end and looked at what the engines actually read. FirstIndex logged 662 answers from ChatGPT, Gemini, Claude, Perplexity and Google AI Mode on November 10, 2026. ChatGPT fetched 2,587 pages while writing its answers, and the five engines cited 5,060 sources between them. Not one was an llms.txt file. The answers cited firstindex.com pages 283 times and never cited firstindex.com/llms.txt, which was live the whole time.

Who reads llms.txt, then?

Barely anyone, and not the ones you’d hope. In the Ahrefs logs, named AI tools made just 19.5% of the visits to the few files anyone fetched. GPTBot, OpenAI’s training crawler, came first. Second was Claude Code, Anthropic’s coding agent, ahead of every AI search and assistant bot. Anthropic keeps an llms.txt for its own developer docs.

So if developers build on your API or SDK, a docs llms.txt has a reader: their coding agents fetch it. A fetch isn’t a citation, though, and a coding agent reading your docs isn’t ChatGPT recommending you to a buyer.

Chrome’s Lighthouse audits the file too, but it only fails a page when fetching llms.txt hits a server error. No file, no penalty.

Is llms.txt like robots.txt or a sitemap?

They get lumped together, but only one of the three can keep you out of AI answers.

What robots.txt, a sitemap and llms.txt each dorobots.txt says which crawlers may fetch which pages; blocking an AI search crawler keeps your pages out of its answers. sitemap.xml lists the pages you want crawled and helps crawlers find them. llms.txt is a Markdown summary of your key pages; it controls nothing, and Google Search doesn’t use it.robots.txtSays which crawlers may fetch which pages.Can keep you out of AI answers.sitemap.xmlLists the pages you want crawled.Helps crawlers find them.llms.txtA Markdown summary of your key pages.Controls nothing.
Sources: Google’s robots.txt and sitemap guides, OpenAI’s crawler docs.

Block OAI-SearchBot in robots.txt and ChatGPT search answers won’t show your pages. Block PerplexityBot, which Perplexity uses to surface and link sites, and it can’t fetch them. llms.txt can’t undo either: it has no rules in it, only links. Our robots.txt checker shows which of 14 search and AI crawlers can read a page on your site.

Should you add an llms.txt file?

Yes. It’s cheap, and we keep one. Just do it last:

  1. Let the AI crawlers inCheck that robots.txt doesn’t block OAI-SearchBot, PerplexityBot or Googlebot.
  2. Get your pages indexedGoogle’s AI Overviews and AI Mode can only show pages that are indexed and eligible to show a snippet.
  3. Get onto the pages AI readsAnswer the questions buyers ask, and get onto the lists and comparisons in your category. Our ChatGPT guide shows which pages it cites.
  4. Then add llms.txtIt takes about ten minutes. Have developer docs? Do this step sooner.

Steps 1 to 3 are where AI visibility comes from. That’s the work we do for 30+ software companies: real changes ship to your site every day, and we check each one before it goes live, for $2,000 a month.

How we counted: FirstIndex’s own panel of 35 questions about hiring an SEO or AI visibility service for a software company, and about FirstIndex, asked 1 to 6 times each on ChatGPT, Gemini, Claude, Perplexity and Google AI Mode on November 10, 2026, with web search available, not forced. 662 answers. We counted every source each answer cited, plus every page ChatGPT fetched (only ChatGPT lists those), and looked for any address ending in llms.txt or llms-full.txt.