I built a content gap workflow that scrapes, clusters, and dumps to a sheet

I built an n8n workflow that finds content gaps automatically: submit a form, it scrapes every page of a site, an AI prompt clusters the results into topics, and a Google Sheet gets created with the findings.

Four steps work end to end. One of them isn’t clean yet.

Here’s the 5-minute walkthrough:

The trigger

The workflow starts from an internal form submission, not a public form tool. I tested it against my own site, which doesn’t have many pages, just enough to prove the concept before pointing it at anything bigger.

Scraping without Firecrawl

I kept hitting Firecrawl’s limits, so I dropped it. Instead I loop over every page on the site directly and pull the raw HTML. From each page I keep only what matters for clustering: the headings and the body content converted to markdown, everything else gets thrown out before the next step.

I’m also evaluating Crawl4AI, an open source crawler, as a possible replacement for the manual loop. That’s a separate experiment though, not part of this workflow yet.

Clustering with a custom prompt

The scraped pages get passed into a prompt I wrote and iterated on with AI help. It clusters the pages by topic, which is what surfaces the actual gaps: topics I don’t have a page for, or topics split across pages that should probably be one. This part works, and I’ve run it enough times to trust the output.

Output: a Google Sheet with a timestamped name

The clustering results and the scraped page data combine into one Google Sheet, named “SEO Content Gaps” plus the date and time it ran, so every run produces a uniquely named file in my Drive. There’s a catch: I can’t create that sheet without pre-setting the filename and a sheet ID first. Both have to be set up manually before I run the workflow, not generated on the fly.

What’s still manual

Here’s the part I’m not smoothing over: the output includes extra rows that shouldn’t be there, and I delete them by hand after every run. The scraping step works. The clustering step works. The output step is the one that isn’t finished.

Where this actually stands: form trigger, page loop, clustering prompt, and timestamped sheet output all run without me touching anything. Presetting the filename and sheet ID, and cleaning junk rows out of the sheet, are still manual.

Shipping a workflow with two manual steps beat waiting until it was fully automated. I’ll fix the output step when it annoys me enough.

Next step

This is the same n8n foundation behind the queue-mode automation platform I run. For the other half of content visibility, see how I optimized this site for LLM search. Want a workflow like this built for your content? It is one of the AI automation systems I build. Book a call.