An Automated Workflow for Clustering Low-Volume Long-Tail Keywords

Ahrefs can hand you 200 long-tail variants of one seed term, each with a search volume too small to justify its own page. Embeddings do the clustering at that scale; the model call only has to name the group and decide what to do with it.

The problem with one keyword, one page

Pull a seed term through Ahrefs' Matching terms report and you don't get ten keywords back. You get a hundred, or two hundred, most of them sitting at 10 to 90 searches a month, each individually too small to justify a dedicated page. Ignore the whole list and you leave real, if scattered, demand on the table. Write a thin page for every variant and you get keyword cannibalization and a content team burning hours on pages that will never individually rank. The actual fix sits between those two bad options: group the variants that share intent, then decide per group whether it earns a new page, folds into one you already have, or isn't worth building at all.

Where a single-call clustering workflow breaks

A keyword-clustering workflow published earlier on this site handles topic grouping with one AI call: paste the whole keyword batch into a single prompt and let the model return topic clusters. That works cleanly up to a few hundred keywords. Past that, you're pushing thousands of tokens through one call, and the model starts missing near-duplicate keywords or splitting groups that should be one. That's not really a clustering problem. It's a context-window problem, and the fix isn't a bigger call, it's not asking an LLM to do the grouping at all.

Step 1: Pull the long-tail variants, filtered to actually small volume

Ahrefs' Keywords Explorer has a Matching terms report built for exactly this: enter a seed term and it returns every keyword phrase containing it, not just close synonyms, and you can set the Volume filter to a low ceiling to isolate the true long-tail phrases the seed term's own volume figure never shows you. Export it as keyword, volume pairs. That's the entire input this workflow needs.

Step 2: Turn every keyword into an embedding, not a prompt

This is the step that avoids the context-window ceiling. OpenAI's text-embedding-3-small model turns each keyword into a 1,536-dimension vector, and a single API call accepts up to 2,048 separate text inputs at once, each capped at 8,191 input tokens, a limit a keyword phrase never comes close to. A list of 500 long-tail keywords costs one API call, not five hundred, and produces a vector for every keyword whose distance to every other vector is just math. No model has to read the whole list at once to group it.

Step 3: Cluster by similarity, without asking a model to read the list

With every keyword now a vector, group them with a standard clustering approach. Agglomerative clustering on cosine similarity works well here, since you don't know the number of clusters in advance the way k-means requires you to. Anything above roughly 0.80 to 0.85 cosine similarity merges into the same cluster; anything below stays separate. This step is pure math, no LLM call, and it's what actually scales past the batch-size ceiling the single-call scenario runs into: clustering a thousand keyword vectors this way costs milliseconds, not tokens.

Step 4: One model call per cluster, checked against your own page inventory

Clustering gets you groups of related keywords. It doesn't tell you what to do with each group. That's a single, small model call per cluster, not per keyword, and it needs your site's own existing content as context, the same pattern an earlier pre-publication keyword-check workflow on this site used: give the model your real page inventory, not just the candidate keyword, so it can tell "this is new" from "this is already covered under a different name."

The schema per cluster:

• cluster_label: a short human-readable name for what the group of keywords is actually asking
• combined_volume: the sum of the individual keyword volumes in the cluster, since that's the number that actually matters, not any single keyword's tiny figure
• verdict: one of new_page, merge_into_existing, or skip_too_niche
• target_page: the slug of the existing page this should merge into, required when the verdict is merge_into_existing, otherwise null
• reasoning: one sentence explaining the call

skip_too_niche is the option a lot of these workflows leave out, and it matters. A cluster of three keywords with a combined volume of 15 searches a month isn't worth a page or a merge, it's worth ignoring, and the schema should let the model say so instead of forcing every cluster into a publish decision.

A worked example

Take a seed term like "ai content brief" run through the Matching terms report. A realistic slice of the long-tail variants it returns: "ai content brief generator" (90/mo), "ai content brief template" (70/mo), "how to write an ai content brief" (40/mo), "content brief automation tool" (30/mo), "automate content briefs with ai" (25/mo), "ai content brief for freelancers" (20/mo), and "content brief generator for agencies" (15/mo). These volumes are illustrative; we don't have a live Ahrefs pull wired into this workflow, only the report and filter it would use.

Embed and cluster that list and you'd expect two groups to fall out: one around brief generation and templates, one around the how-to and automation angle, with the agency- and freelancer-specific phrases likely folding into whichever cluster they sit closest to rather than standing alone.

Now run the routing call with this site's own real inventory as context. This site already has two posts substantially covering content-brief generation and automation: "A Multi-Agent Workflow for SEO Content Brief Generation" and "How to Automate SEO Content Briefs Without Losing Brand Voice." A cluster built from those seven keywords should come back merge_into_existing, not new_page, pointed at whichever of those two posts covers the closer angle. Skipping the inventory-check step here would have been a real mistake: a workflow that only sees the seven keywords, with a combined volume north of 250 searches a month, would look exactly like a new_page case on the numbers alone, and would have shipped a third, redundant post on a topic already covered twice.

Where this breaks

• The similarity threshold isn't universal. Pick it too loose and you merge keywords with genuinely different intent into one cluster; pick it too tight and near-duplicates that should obviously be one page stay split. There's no default that's right for every seed term. Run a first batch, spot-check the clusters by eye, and adjust before trusting the threshold at scale.
• The inventory check only catches topic-level overlap. It won't catch a page that answers the same question under a title or framing the embeddings and the model both miss. Treat a merge_into_existing or skip_too_niche verdict as a strong default, not a final answer, especially early on.
• Volume numbers are a snapshot, not a live feed. Whatever you pull from the Matching terms report is accurate the day you pull it. Long-tail demand shifts, and a cluster too small to bother with in January can be worth a page by summer. Rerun the pull periodically instead of treating one export as permanent.

The three-question recipe

If you're building this for your own long-tail keyword pile: what's your volume ceiling for "too small to stand alone" (start with 10 to 100 a month and adjust), what's your similarity threshold for merging keywords into one cluster (start around 0.80 to 0.85 cosine similarity and spot-check), and what's the one schema, with a real skip option, that turns a cluster and your own page inventory into a publish decision. That's the whole workflow, and the expensive part, reading through a keyword list, never touches an LLM.