keyword_clustering v1.0.0

name: "keyword_clustering" description: "Group keywords into topic clusters for content planning: collect the raw list, cluster by search intent and SERP overlap instead of wording, name one pillar page per cluster, and run a gap analysis. Use when someone has a keyword export to organize, a content plan to build, pages that compete with each other, or a pillar and cluster architecture to design."

Keyword Clustering

Purpose

Turn a flat keyword export into a content plan a writer can execute. The output is a cluster map: one pillar page per topic, one page per distinct intent under it, internal links between them, and a gap list that says what is missing. Clustering by intent is the whole point. Two phrasings that look different can want the same page, and two phrasings that share most of their words can want different pages.

Workflow

1. Collect the raw list with its numbers. Pull keywords from the research tool export, Search Console queries, competitor exports, and questions from People Also Ask and support tickets. Keep volume and difficulty when you have them. Drop exact duplicates and obvious junk, but do not prune by gut feeling yet. Clustering needs the messy list, because the mess is where the topics hide. Save it as a CSV with columns keyword, volume, difficulty, serp_urls, url. Fill url only when a live page already covers that keyword.

1. Classify intent for every keyword, one intent per keyword. Use the four standard practitioner intents. Informational means the searcher wants to learn (how, what, why, guide, tutorial). Navigational means they want a known brand or page (brand name, login, a specific product page). Commercial means they are comparing before buying (best, vs, review, top, alternatives). Transactional means they are ready to act (buy, price, hire, near me, book). When modifiers conflict, the stronger buying signal wins: "best" plus "price" is transactional, not commercial. Treat the modifier list as a first pass. Spot-check the top results for any keyword you are unsure about, because the results page shows what the search engine thinks the job is.

1. Run the bundled cluster tool for the first pass. Run python3 bin/cluster_keywords.py keywords.csv --out-md map.md --out-csv clusters.csv. The tool groups by intent first, merges keywords whose top results share at least 3 URLs when serp_urls data is present, then groups the rest by parent topic (the keyword with intent modifiers and filler words stripped) plus intent. It scores each cluster with opportunity equals total volume divided by average difficulty plus 1, and flags thin clusters and clusters with no live URL.

1. Adjudicate the gray zone by hand. The tool is a first pass, not a verdict. Review three cases. Near merges: clusters the tool split on wording where both sides want the same job, for example a how-to phrasing and a guide phrasing for the same task. Merge them, and demote the weaker phrasing to a section heading. False merges: keywords the tool grouped that want different formats, for example a ranked best list and a head-to-head comparison. Split them and link them to each other. Mixed SERP cases: when shared top URLs sit at 1 or 2, open the results and decide from the formats ranking, not from the words in the query. Record every merge and split decision in the map, with the reason in one line.

1. Name the pillar page for each topic. Group the finished clusters under topics. The pillar keyword is the broad head term for the topic, usually the shortest phrasing with real volume. The pillar page covers the whole topic at overview depth, carries a table of contents, and links to every cluster page under it. A topic earns a pillar only if it supports at least four or five distinct cluster pages with distinct intents or subtopics. Below that, write one strong page and skip the pillar structure. Write the pillar page name as a promise a reader would recognize, not as the raw keyword with words bolted on.

1. Assign one page per cluster and kill cannibalization. Each cluster gets exactly one owner page and one primary keyword. Every other keyword in the cluster is a secondary phrasing that page must naturally cover. If two planned pages would satisfy the same searcher equally well, you have cannibalization: merge them, or retarget one to a clearly different intent. If an existing site already has two pages on one intent, the fix is the same: pick the stronger page, fold the useful material from the weaker one into it, and redirect the weaker URL. Mentioning the same phrase on two pages is fine when the intents differ, for example a how-it-works guide and a best-of list for the same category.

1. Map the internal links before anyone writes. Every cluster page links back to its pillar with descriptive anchor text. The pillar links out to every cluster page. Cluster pages link sideways to the nearest related cluster page where a reader would naturally want it, such as a feeding guide linking to a buy page for a starter that failed. Write the link map into the deliverable as a list, pillar first, so the writer and the developer build the same structure.

1. Run the gap analysis. Check each topic for four gaps. Missing intent: a topic with informational pages and no commercial or transactional page gives learners no next step. Missing questions: question phrasings in the export with no cluster to hold them. Competitor gaps: keywords and subtopics competitors rank for that have no cluster in your map, taken from the competitor export you collected in step 1. Thin clusters: one-keyword clusters that signal either a merge candidate or a list that is too small to plan from. Report gaps as decisions, not trivia. Each gap gets one line: what is missing, why it matters, and whether you recommend filling it now, later, or not at all.

1. Prioritize and sequence the build. Sort clusters by the opportunity score, then adjust by hand for business relevance. A lower-volume transactional cluster that matches what the business sells usually beats a high-volume informational cluster that does not. Mark each cluster as net-new or refresh based on the url column. Sequence refreshes of pages that already get impressions before net-new pages in crowded topics, unless the gap analysis says the topic has no conversion page at all.

Output Contract

Deliver a cluster map document containing:

• The input summary: how many keywords went in, where they came from, and what thresholds the tool used.
• One section per topic: pillar page name, target keyword, intent, URL slug, total volume, and opportunity score.
• Under each pillar, every cluster page with its primary keyword, intent, secondary phrasings, content format (guide, comparison, product page, FAQ), and status: net-new or refresh of a named URL.
• The link map: pillar to cluster links, cluster to pillar links, and the sideways links, listed explicitly.
• The adjudication log: every merge and split from step 4 with a one-line reason.
• The gap analysis: each gap with a now, later, or skip call.
• A build order: clusters sorted into first, next, and later, with the reason for the first three picks.

The worked example in references/cluster-map-example.md shows this contract filled in on a small list.

Operating Rules

• One intent, one page. This rule outranks volume. Never brief two pages for the same intent because both phrasings have search volume.
• The four intent labels are a practitioner diagnostic, not a taxonomy published by a search engine. Verify intent against live results whenever a keyword matters and the modifiers are ambiguous.
• SERP overlap thresholds in the tool (3 shared top URLs by default) are working defaults for this method, not industry standards. State the threshold you used in the deliverable so the next person can reproduce the map.
• Never invent volume or difficulty numbers. If the export has no numbers, cluster on intent and topic alone and say the priority order is judgment, not data.
• A cluster map is a plan, not published content. Do not present planned pages as live. Check the url column against the real site before calling anything a refresh.
• Keep clusters tight enough to brief. A cluster page should answer one job well. If a cluster needs two different formats to satisfy it, it is two clusters.
• Revisit the map when the keyword list changes materially or when Search Console shows two of your pages trading places for the same query, which is cannibalization showing up in live data.