An n8n workflow that scrapes a site, clusters its pages with an AI prompt, and writes the gaps to Google Sheets. What works, and what still needs cleanup.
Claude Code saw duplicate content, applied the textbook fix, and broke every TIL post on this site. The bug wasn't the model. It was where I kept my decisions.
Generative engine optimization with Claude Code: dynamic llms.txt endpoints, AI-crawler rules in robots.txt, and JSON-LD on every page. One session.
No paid crawler, no SaaS audit tool. Claude Code crawled the built site with grep, found three 404s, traced the root cause, and shipped a custom 404 page.
A static llms.txt goes stale. An Astro endpoint that reads your content collections rebuilds it every deploy. One gotcha: public/ silently shadows it.
TIL posts lived in their own Astro content collection. Folding them into the blog collection as a category field fixed SEO and cut duplicate logic.
