What is llms.txt?
llms.txt is a short Markdown file at the root of your site, at /llms.txt, that gives AI tools a curated map of your most useful pages: your name, a one-line summary and lists of links with notes. It was proposed by Jeremy Howard in September 2024 and is documented at llmstxt.org.
Be clear about what it is: a proposal, not a standard. None of the major AI providers has committed to reading llms.txt when its assistants or crawlers look at your site, and Google says Google Search ignores it, so it doesn't affect rankings, AI Overviews or AI Mode. Some coding assistants and agents read it on documentation sites, and Chrome's Lighthouse includes an llms.txt check that only flags server errors when fetching the file.
Treat it as a low-cost extra: if you already have clear pages, writing one takes minutes and does no harm. It isn't a ranking lever. What gets you cited in AI answers is crawlable, well-structured content; see GEO services.
How llms.txt differs from robots.txt and sitemap.xml
- robots.txt is a widely followed standard (RFC 9309) that tells crawlers what they may fetch. It grants or withholds permission; it doesn't describe your content.
- sitemap.xml lists every URL you want search engines to discover, for machines only, with no descriptions.
- llms.txt is a short, human-readable guide to your best pages, with a sentence on each, written for language models to read in one go.
They work together: robots.txt must still allow the AI crawlers you care about (check it with the AI Crawler Checker), and the generator here builds llms.txt from your sitemap.
What is llms-full.txt?
llms-full.txt isn't part of the llmstxt.org proposal. It's a convention used mostly by documentation sites: one file at /llms-full.txt with the full text of the pages llms.txt links to, so a tool can read everything without following links. Cloudflare and Svelte publish one for their developer docs.
For most business sites it isn't needed. The proposal instead suggests offering Markdown versions of useful pages at the same URL with .md added. The validator notes when llms-full.txt is missing, as a suggestion only.
What the validator checks
Every rule comes from the format at llmstxt.org:
- Exactly one H1, the site or project name, first in the file (the only required part).
- An optional blockquote summary straight after the H1.
- Details without headings, then H2 sections of links written as
- [name](url): notes. - An optional section named exactly “Optional” for links an agent can skip.
- Malformed links, relative or non-web URLs, duplicates, empty sections and the first 25 links' health (broken, redirecting or unreachable).
- Served at the root with HTTP 200, as plain text or Markdown in UTF-8, at a sensible size.
Errors break the format and cap the grade at C; warnings and suggestions improve how well agents can use the file. The cleaned-up file fixes everything that can be fixed automatically.
Where to upload llms.txt
The file must be reachable at https://yourdomain.com/llms.txt:
- Vite, Next.js and most static sites: put
llms.txtin thepublicfolder; files there are served from the site root (Vite, Next.js). - WordPress: upload it to the folder that holds
wp-config.phpwith your host's file manager or SFTP, or let an SEO plugin create it, such as Yoast SEO's llms.txt feature. - Shopify: you can't upload files to the store root, and Shopify already serves an automatic
/llms.txtwith shopping instructions for agents. Validate what yours returns before changing anything; apps such as Yoast SEO for Shopify can manage it. - cPanel hosting: open File Manager, go to
public_htmland upload the file there.
Then validate the live URL here. If you'd rather we set it up with the rest of your AI search optimisation, send us the task.
FAQ
llms.txt Validator & Generator questions
Not on its own. It's a proposal, not a standard, and Google says Google Search ignores it, including AI Overviews and AI Mode. No major AI provider has committed to reading it. It's a low-cost extra, not a ranking lever.
None of them has said its crawlers or assistants use llms.txt when answering questions about your site. Some coding assistants and agents read it on documentation sites, which is where it's most useful today.
No. robots.txt controls which crawlers may fetch which pages. llms.txt doesn't grant or block anything; it's a curated summary of your best pages. You still need robots.txt to allow the AI crawlers you want.
Usually not. It isn't part of the llms.txt proposal; it's a convention for documentation sites that want their full content in one file. Most business sites only need llms.txt.
Any error means the file doesn't follow the format yet, for example a missing or second H1, or a list item that isn't a valid Markdown link. Fix the errors, or copy the cleaned-up file, and re-check.
It reads your sitemap (from robots.txt, else /sitemap.xml), fetches each page's title and meta description, and groups pages by the first part of their URL. You choose the sections, order and pages before copying the file.
No. Files are fetched and checked on request and the results stay in your browser. Your details are only saved if you ask for the PDF report.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the llms.txt Validator & Generator found and make the rest of your site easy for Google and AI answers to understand.
