Building llms.txt as an Astro endpoint so it never goes stale

llms.txt is the convention for telling AI crawlers what lives on your site. Think of it as a markdown sitemap written for models instead of browsers. My first instinct was the obvious one: drop a file in public/ and move on.

That instinct was wrong, and I want to save you the rebuild it cost me.

A file in public/ is frozen the moment you write it. Every post I published after that point left the index a little more out of date, and I never once remembered to go back and hand-edit a text file. Two weeks in, my llms.txt described a site that no longer existed.

The fix on Astro is to stop treating it as a file and make it an endpoint that reads from your content collections at build time:

// src/pages/llms.txt.ts
import { getCollection } from 'astro:content';
import type { APIRoute } from 'astro';

export const GET: APIRoute = async () => {
  const posts = await getCollection('blog', ({ data }) => !data.draft);
  const lines = posts.map((p) => `- [${p.data.title}](https://example.com/blog/${p.id}/): ${p.data.description}`);
  return new Response(`# Site\n\n${lines.join('\n')}`, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
};

On a static build, Astro renders this to dist/llms.txt like any other route. Every deploy now ships a complete, current index with zero maintenance. Add a post, run the build, the index already knows about it.

Here is the gotcha that actually cost me the rebuild: if public/llms.txt still exists, it shadows the endpoint. Astro copies everything in public/ over the build output, so the stale file wins and your shiny new endpoint silently never serves. Delete the old static file first, then verify with curl yoursite.com/llms.txt that you are getting the generated list and not the frozen one.

The same pattern scales straight to llms-full.txt. Return post.body for each entry instead of the description, and a model can ingest your entire site in a single fetch. That is the version I actually ship now, because if I am going to make the site legible to LLMs, I would rather hand them the whole text than a table of contents.

Next step

This endpoint was one part of a larger pass: the full GEO setup for ChatGPT, Claude, and Perplexity. If you want your own site legible to LLMs, a consulting session is enough to map it. Book a call.