How Agencies Add llms.txt to Client Sites Without Breaking Anything
If you run an agency, “AI readiness” is turning into a line item clients ask about. llms.txtis a concrete, deliverable answer — but doing it across a portfolio of clientdomains has a risk doing it on your own site doesn't: you are publishing content at a domain you don't own, that a machine will quote. Get a title or a claim wrong and it is your client's brand, not yours.
1. Scope to public HTTPS pages only
Never point a crawler at authenticated or staging URLs on a client site. Restrict to public pages. This alone avoids the most common way these projects go sideways.
2. Inventory before you draft
Pull the eligible public URLs with their real titles and descriptions. You want a page inventory you can hand the client — it doubles as a mini content audit and often surfaces stale or duplicate pages they didn't know were live.
3. Draft a navigable map, not a data dump
A good llms.txt is short and readable: what the site is, then the pages that matter, each with a one-line description. Add llms-full.txt only when the content volume justifies it.
4. Review before it touches the client's root
This is the step that separates a professional handoff from a liability. Flag ambiguous or off-brand titles, sensitive or legally risky claims, pages that should be excluded, and anything a machine shouldn't repeat verbatim. Your client approves, then it publishes at clientdomain.com/llms.txt.
5. Set expectations honestly
Tell clients the truth: llms.txt helps AI systems readthe site accurately. It does not promise rankings, citations, or traffic. Selling it as a ranking hack burns trust when the numbers don't move.
Don't want to build the pipeline yourself?
Standing up this inventory-to-review pipeline for every client is real work. If you would rather white-label the delivery, Site Context Forge runs it as a bounded batch — up to 5 client sites, 3 business days, page inventory plus llms.txt plus editorial review notes, $1,500 fixed, 50% deposit. Reviewed before delivery, no ranking promises.
See also: Do You Need an llms.txt File? and llms.txt vs robots.txt vs sitemap.xml.