llms_txt_generation v1.0.0

name: llms_txt_generation description: "Write and maintain an llms.txt file a site can stand behind: select the pages an agent should read, write short notes checked against each live page, keep private and thin pages out, and recheck the file when the site changes. Use when someone asks to create, audit, fix, or update an llms.txt or llms-full.txt file, or asks whether their site needs one."

llms.txt Generation

Purpose

Produce one Markdown file at /llms.txt, or at a subpath such as /docs/llms.txt, that gives an agent a short, accurate map of a site: what the site is, which pages matter, and what each page covers. The value is in the checking, not the file. A stale or padded llms.txt teaches agents wrong facts with confidence, which is worse than no file at all.

Be plain about the limits in every delivery. The format is a community proposal from llmstxt.org, first written in 2024 and now at version 2. Google Search Central states in its generative AI guide that Google Search does not use llms.txt or other new machine readable files to appear in Google Search, including its generative AI features. No checked primary source says a major answer engine cites a page because of this file. The proven use is narrower and real: documentation and API sites that coding agents and docs tools read on demand, plus agents the site owner points at the file directly.

Workflow

1. Decide if the site should have one at all. Say yes when the site has documentation, API reference, guides, pricing, or policy pages that people already paste into assistants by hand, or when agents the owner controls need a stable map. Say it is optional, low value work for a small brochure site with no docs, and say so before spending time on it. Never frame the file as a ranking or citation tactic.

1. Run the page selection worksheet in references/page-selection-worksheet.md. Start from real questions the site must answer, pull candidates from navigation, traffic, repeated support questions, and the docs the team shares by hand, then apply the cut list: thin pages, gated or private pages, stale pages, claim only pages, drafts, staging URLs, and duplicates all come out. One row per candidate, with the question it answers written down. A page with no question is out.

1. Fetch and read every kept page live before writing its note. Follow references/accuracy-check.md: open the live URL, confirm it loads, read the title, first heading, and first section, and note prices, dates, limits, and audience exactly as the page states them. Never write a note from the page title, from memory, or from a sitemap description.

1. Draft the file in the exact order the spec sets out. One H1 with the site or project name, the only required element. A blockquote right after it with the key facts an agent needs to understand the rest, each clause checked against a public page. Detail paragraphs or lists next, with no headings in that zone, covering how to interpret the links, ordering rules, account requirements, and common traps. Then H2 sections of file lists, each item in the form - Title: short note. Put secondary links under a final H2 titled Optional. Full format and source notes are in references/spec-and-limits.md, and a complete file for a small site is in references/worked-example.md.

1. Write notes that an agent can act on. One short sentence or fragment per link. Say what the page does, who it is for, and any limit that changes the answer, for example pickup only, account required, or prices effective on a stated date. Skip praise words and skip repeating the site name. If the page cannot support the note you wanted to write, fix the note, and flag the page gap to the owner as separate work.

1. Validate the file. Run python3 bin/llms_check.py <path-to-llms.txt> and fix every error. Then run it again with --fetch to catch dead links, home page redirects, and link text that no longer matches the page title. The fetch pass does not certify a note. The per link accuracy table in references/accuracy-check.md must be complete before the file ships, with a row for every link.

1. Publish and wire up discovery. Place the file at the root, or at the subpath it covers, served as a plain Markdown text file with a 200 status at the exact path. Verify with curl after publishing, not from the build output. Where the platform allows it, add the version 2 link relations: rel="describedby" pointing to the llms.txt from the pages it covers, and rel="alternate" type="text/markdown" pointing to clean Markdown versions of pages where they exist, as an HTML link element or an HTTP Link header. If a generator plugin already maintains the file, check what it emits against steps 3 to 6 instead of hand editing its output, and record where the true source lives.

1. Handle llms-full.txt only when a named tool needs it. It is a platform convention, not part of the spec checked for this skill. Generate it from the same page sources as llms.txt, in the build, and never maintain it by hand as a second file.

1. Set the recheck before you finish. Keep a change log beside the file in version control, following references/worked-example.md. Log each change, the pages rechecked, and the next trigger. Triggers: a listed page edited, moved, or removed, a price or hours change, a new main section, a change to any fact in the blockquote, or a fixed review interval for quiet sites. On a trigger, rerun steps 3, 6, and 7 for the affected links, not just the edited one.

1. Test the file the way the spec suggests. Give an agent only the llms.txt and ask it five real questions from the selection worksheet. It should answer from the file and one follow up fetch per question. Where it guesses, picks the wrong page, or quotes a fact the file got wrong, fix the file and test again.

Output Contract

• The llms.txt file itself, in spec order, validated by bin/llms_check.py with zero errors.
• The completed page selection worksheet, showing kept, Optional, and cut pages with reasons.
• The accuracy table: one row per link, with the live page title seen and the date that link note was checked against the page.
• A change log entry for this version, plus the named recheck triggers.
• A short limits note to the owner, in plain words: what this file does, that Google Search ignores it per Google Search Central, and that no citation or ranking gain is promised.

Operating Rules

1. Every factual claim in the file must be verified against the live page it describes, in this run. Unverifiable means cut it or leave it out. No numbers, prices, dates, or quotes from memory, ever.
2. Never list a private, gated, thin, or stale page to make the file look complete. A shorter file that is right beats a long file that is partly wrong.
3. Never promise search ranking, AI Overview, AI Mode, or citation benefit. If the request rests on that promise, correct it at the start and let the owner decide whether the narrower benefit is worth it.
4. Keep the file curated. Sitemaps list everything, robots.txt controls access, this file does neither. If the draft reads like a sitemap, go back to step 2.
5. No em dashes in the file or in any note. Write plain sentences, keep notes short, and cut any adjective the linked page does not prove.
6. Publish nothing until the accuracy table is complete. An unchecked link is a draft link, and draft links do not ship.