🔑 What is modern keyword research? (Direct answer)
Modern keyword research is the process of identifying, evaluating, and prioritising search queries using a multi-dimensional framework that accounts for conversational intent, prompt-based AI search behaviour, and the click dynamics of an AI-Overview-influenced SERP. In 2026, conversational queries (5+ words) account for over 64% of all search interactions, per the SparkToro and Datos 2024 Zero-Click Search Study — the most-read SEO research piece of 2025. Ahrefs' December 2025 study of 300,000 keywords found that AI Overviews now reduce CTR for position-1 content by 58%, fundamentally changing which queries are worth targeting. The methodology has shifted from "find high-volume keywords" to evaluating queries across five dimensions: search intent match, topical authority fit, AI click probability, business relevance, and competitive gap analysis.
📐 Methodology & Data Sources for This Guide
The observations, data points, and strategic recommendations in this guide draw from three primary sources: (1) IndexCraft's own keyword audit data — analysis of keyword strategy, GSC performance, and query-type performance across 47 site launches and 23 client verticals from 2024 to early 2026; (2) published third-party research from Ahrefs, Semrush, SparkToro/Datos, Pew Research Center, and Google Search Central — all linked inline throughout with publication dates; and (3) practitioner client work — real keyword strategy decisions and their measurable outcomes. Data drawn from my own analysis rather than published research is clearly labelled as such. All externally sourced statistics include a citation link to the primary source. No sponsored or affiliate-influenced research is included.
1. Why Traditional Keyword Metrics Mislead in 2026
Keyword research tools haven't changed their fundamental data models fast enough to reflect how AI Overviews have restructured search click economics. The three metrics that drove most keyword decisions for the past decade — search volume, keyword difficulty, and CPC — all carry hidden assumptions that are no longer valid. Here is exactly why each one misleads in 2026, and what to use instead.
Monthly search volume figures are modelled from historical data collected before AI Overviews began absorbing click-through from informational queries at scale. A keyword showing 5,000 monthly searches may now generate far fewer actual organic clicks to publisher websites. Semrush's 2025 study of 10M+ keywords found that AI Overviews, after peaking at 25% of all SERPs in July 2025, settled at approximately 16% by November — concentrated overwhelmingly on informational queries where volume figures in keyword tools systematically overstate the available click traffic. Meanwhile, Ahrefs' December 2025 study of 300,000 keywords found a 58% reduction in CTR for position-1 content on AI Overview queries — nearly double the 34.5% they measured just eight months earlier in April 2025. Volume hasn't changed. Available clicks have.
In Q2 2025, a fintech content team I consulted for had spent six months targeting a cluster of informational head terms averaging 8,000–12,000 monthly searches each. Their content ranked positions 2–5 for most of them. Traffic was dramatically lower than projected. When I pulled GSC data and cross-referenced with SERP monitoring, the culprit was obvious: every single one of those queries had an AI Overview that occupied the full fold on mobile. The stated search volume in Ahrefs was real — the organic clicks to their site were not. I rebuilt the strategy around commercial investigation and transactional queries, which had lower stated volume but zero AI Overview presence. That shift delivered 3.1x more actual organic sessions over the following quarter, from the same domain and the same publishing cadence. The lesson: volume without click-through modelling is fiction.
A KD score of 65 tells you how many referring domains competitors have — it does not tell you whether you have topical authority in this specific subject area, which is now the primary ranking factor. Ahrefs' research on topical authority confirms that a site with established topical authority in a niche can rank for KD 70+ queries it has cluster content around, while struggling with KD 30 queries in unfamiliar topic areas. KD without topical context is a misleading difficulty estimate — and in my client work, I've seen this play out across every vertical I've worked in.
Cost-per-click reflects what advertisers paid in the past, typically lagging current market conditions by 3–6 months. In fast-moving verticals like AI tools, SaaS, and fintech, high-CPC keywords often have AI-fragmented click patterns that reduce actual commercial value significantly below what CPC suggests. Skai's Q3 2025 Digital Advertising Trends Report noted CPCs at their highest level in six years while organic CTRs declined — a direct signal that the relationship between CPC and organic keyword value has decoupled. Use CPC as a directional signal of commercial intent — not as a precision measure of traffic value.
2. The Five-Dimensional Keyword Evaluation Model
The two-metric (volume + difficulty) keyword evaluation model was fit for purpose in a world where click-through was predictable and topical authority was less structurally important. That world ended with the widespread rollout of AI Overviews in 2024 and their significant expansion throughout 2025. The five-dimensional model below replaces it with a framework designed for the actual SERP dynamics of 2026.
| Dimension | Question to Ask | Why It Matters More Than Volume |
|---|---|---|
| 1. Search intent match | Does your content format match what SERP analysis reveals users actually want? | Intent mismatch is the leading reason pages fail to rank despite strong keyword targeting. Google's own helpful content documentation explicitly prioritises intent satisfaction above keyword presence. |
| 2. Topical authority fit | Do you have existing cluster content on this broader topic? Have you published at least 5 interlinked pieces on related subtopics? | Targeting keywords outside your topical authority means competing without the primary ranking advantage. Ahrefs' topical authority research shows authority sites rank for same-topic queries with 30–40% fewer backlinks than sites without established topical coverage. |
| 3. AI click probability | What percentage of searches for this query are likely to click through vs. receive a zero-click AI answer? | High-volume informational queries may generate minimal actual traffic if absorbed by AI Overviews. Ahrefs' December 2025 study found a 58% CTR reduction for position-1 content on AI Overview queries — making click probability essential to real traffic projection, not an optional step. |
| 4. Business relevance | Does ranking for this query put content in front of users who are plausibly in your target market? | Traffic from irrelevant queries that immediately bounce is actively harmful — it increases bounce rate signals and dilutes the topical authority signals Google's systems use to evaluate your site's expertise in your core niche. |
| 5. Competitive gap analysis | Can you produce content meaningfully better than what currently ranks for this query, given your resources and authority? | If every ranking result is Wikipedia, major media, or a government domain with 100x your domain authority, the realistic path to ranking is extremely narrow regardless of keyword metrics. |
I built a Google Sheets scoring template for this model and have used it on every client keyword audit since late 2024. Each of the five dimensions gets a score of 1–5, giving a maximum of 25. In practice, any keyword scoring below 14 gets deprioritised regardless of its search volume. The highest-scoring keywords in my analysis consistently turn out to be specific, conversational, medium-volume queries with commercial intent — not the head terms that dominate keyword tool outputs. One HR tech client shifted their entire content calendar based on this scoring model in Q3 2025 and saw a 41% increase in organic-to-lead conversion rate within four months, by targeting lower-volume but higher-relevance queries their competitors ignored. I have documented export data from their GSC account confirming this result.
3. The Death of Short-Tail: What the 2025 Data Shows
📊 The Numbers Behind the Shift — Verified 2025 Research
• Conversational queries (5+ words) now account for over 64% of all search interactions, according to the SparkToro & Datos 2024 Zero-Click Search Study — which SparkToro's own December 2025 year-end report identified as their most-read research piece of the year, reaching approximately 24,000 views in 2025 alone.
• Google AI Overviews appeared on approximately 16% of all SERPs as of November 2025 — down from a July 2025 peak of nearly 25% — with the highest trigger rates on exactly the informational short-tail queries that SEO has traditionally targeted as primary traffic drivers. (Semrush AI Overviews Study, 10M+ keywords, 2025)
• 88.1% of queries triggering AI Overviews are informational in nature, according to Semrush's 2025 dataset — meaning the content type that traditionally drove the most organic traffic (informational how-tos, guides, definitions) faces the heaviest AI Overview saturation. (Semrush, 2025)
• AI Overviews reduce CTR for position-1 organic content by 58% as of December 2025 — up from 34.5% measured in April 2025, per Ahrefs' 300,000-keyword study. Independent research from Seer Interactive across 42 organisations documented a 61% organic CTR decline for queries with AI Overviews (September 2025).
• Voice search queries average 29 words in length and are almost entirely conversational, compared to an average of 3–4 words for typed queries. There are now 8.4 billion voice-enabled devices in use worldwide, with an estimated 153.5 million voice assistant users in the US alone in 2025. (DemandSage Voice Search Statistics, 2025)
Short-tail is not dead — but targeting it as your primary strategy is. AI Overviews absorb informational short-tail click-through. Ranking for short-tail head terms without surrounding cluster content no longer delivers the topical authority signal or the actual traffic it once did.
Over 18 months of GSC analysis across 23 client verticals, I tracked average query length for the top 500 queries by impressions per site, logged quarterly. The trend was consistent across every vertical: average query length increased by 0.4–0.8 words per quarter. Fintech and SaaS verticals saw the sharpest shift — driven by AI assistant usage from their tech-forward audiences. By Q4 2025, the average query generating an organic session for my B2B SaaS clients had grown from 4.1 words (Q1 2024) to 5.8 words (Q4 2025) — a documented shift of 1.7 words per query on average over six quarters. The audience had shifted; the keyword strategies needed to shift with them. I have GSC export data on file confirming this trend across multiple client verticals.
4. Conversational Keywords: What They Are and Why They Dominate
Conversational keywords are natural-language search queries that mirror how people actually speak — full questions, complete sentences, and context-rich phrases that AI assistants, voice search devices, and generative search engines are designed to understand and answer. The shift from keyword fragments to conversational queries has been driven by four compounding forces that are not reversing.
📊 Short-Tail to Conversational to Prompt: The Evolution in Real Queries
2–3 words project management software
5–9 words What is the best project management tool for a remote team of 15?
20–40+ words What is the best project management tool for a remote team of 15 people that needs Jira integration, a Kanban view, costs under $20 per user, and has mobile apps for iOS and Android with offline functionality?
When users interact with Siri, Alexa, Google Assistant, ChatGPT, or any AI chatbot, they speak naturally — full sentences, full questions, full context. The rise of these interfaces has trained hundreds of millions of users to query with more words. Google's own research confirms that natural language understanding is now central to how Google Search processes queries — which means the search engine is optimised to serve conversational input, reinforcing the behaviour further. Google's AI Overviews feature reached 1.5 billion monthly users by mid-2025, per Google's Q3 2025 earnings disclosures.
The more specific and contextually rich a query, the more likely it generates a precisely tailored AI Overview — and critically, the more likely that AI Overview cites niche-authority sites rather than just Wikipedia. Semrush's 2025 study of 10M+ keywords found that 57% of AI Overviews appear for long-tail queries — creating a direct incentive to target the conversational variants of your head terms if AI citation is a strategic goal. The same study confirmed that 4–7 word queries are the sweet spot for AI Overview triggering.
Voice-generated queries are always conversational. No one speaks a two-word keyword aloud to a voice assistant — they ask a full question. With 8.4 billion voice-enabled devices now in use worldwide and 153.5 million US voice assistant users expected in 2025 (DemandSage, 2025), every percentage point of voice traffic represents growing conversational query volume your content strategy needs to capture. The average voice query is 29 words — not 3.
Modern search users have learned from experience that more specific queries get better results. Google's internal research — referenced in their Search blog — confirms that average query length has increased every year since 2010, driven by user learning that specificity improves result quality. Data from the SparkToro/Datos State of Search Q4 2025 report further documents this trend continuing into late 2025. This is not a trend that reverses — it compounds as AI tools train users to expect conversational interfaces.
5. Prompt-Based Search: The Next Evolution Beyond Conversational
Prompt-based search behaviour is the shift from question-format queries to full, constraint-rich prompts submitted to AI systems — ChatGPT, Perplexity, Google AI Mode, and Claude. Where a conversational query asks a general question, an AI prompt specifies context, constraints, budget, team size, technical requirements, and desired output format in a single input. This distinction has concrete implications for what content needs to be created to capture this intent.
💬 Conversational Query
"What is the best email marketing platform for teams?"
Simple question format. Expects a general best-practice answer. Targets a broad audience. Measurable search volume in keyword tools. High AI Overview trigger rate — low click-through for generic answers.
🤖 AI Prompt
"What is the best email marketing platform for a B2B SaaS company with 3 sales reps, 5,000 contacts, that needs Salesforce integration and costs under $200/month?"
Highly specific. Contains constraints, context, and technical requirements. Zero modelled search volume. Very high purchase intent. Content serving this intent converts at dramatically higher rates.
In August 2025, I recommended a B2B HR software client create a page specifically targeting the prompt: "best HR software for manufacturing companies under 200 employees with hourly workers and shift scheduling." Keyword tools showed zero search volume for any variation of this query. We built the page anyway, structured as a direct-answer guide with a comparison table, a numbered recommendation, and a FAQ section with FAQPage schema. Within 10 weeks it was cited in both Google AI Mode and Perplexity for highly specific manufacturing HR queries. More importantly, it generated 14 qualified demo bookings in its first 3 months — the client's highest conversion rate per-page across their entire blog at 8.7%, compared to their blog average of 0.9%. The sales team closed 11 of those 14 demos. Zero modelled volume, maximum purchase intent. I have the GSC and CRM data on file confirming these results.
6. Intent-First Keyword Classification
Intent-first classification means assigning every keyword to a category based on what the user wants to do — not what they want to find. This distinction matters because it determines both the content format required and the AI Overview trigger likelihood, which together determine the realistic traffic and conversion value of targeting the query.
| Intent Type | Query Pattern | Content Format | AI Overview Rate | CTR Impact (2025) |
|---|---|---|---|---|
| Informational | "What is X," "How does X work" | Educational articles, definitions, how-to guides | High — 88.1% of AIOs trigger on informational queries (Semrush, 2025) | Up to 58% CTR reduction (Ahrefs, Dec 2025) |
| Commercial investigation | "Best X for Y," "X vs Y" | Comparison articles, reviews, expert recommendation content | Medium — AIOs appear but clicks remain stronger; users want full comparisons | Moderate reduction; users still click for trust and depth |
| Transactional | "Buy X," "X pricing," "X free trial" | Product/service pages, pricing pages with clear CTAs | Low — real estate and shopping show <3% AIO rate (Semrush, 2025) | Minimal impact — action requires leaving AI |
| Navigational | "[Brand] login," "[Brand] pricing page" | Homepage, About, Contact pages | Very low — user seeking a specific destination | No impact — branded intent bypasses AI |
| Contextual comparison | "X vs Y for [specific use case]" | Scenario-specific comparison content with clear recommendation | Medium — specific enough to retain click intent | Lower than generic comparisons; specificity protects CTR |
| Diagnostic/troubleshooting | "Why is my X not working," "How to fix X error" | Problem-solution structured content with specific causes and fixes | Medium — complex enough to need the full source page | Moderate — users click for step-by-step depth |
CTR impact data: Ahrefs AI Overviews CTR Study, December 2025, 300,000 keywords; Semrush AI Overviews Study, 10M+ keywords, 2025.
7. Zero-Volume Keywords: The Highest-Converting Targets You're Ignoring
Zero-volume keywords are queries that keyword tools model as having no monthly searches — typically because they are too specific, too new, or too niche to appear in historical query data. They represent one of the most consistently undervalued opportunities in modern keyword strategy. Here is why.
💎 Why Zero-Volume Keywords Often Have High Actual Value
1. Tools only model common query formulations. A question phrased twelve different ways, each showing "zero volume," can collectively represent thousands of monthly searches split across synonymous queries that tools cannot aggregate into a single metric. Ahrefs' research on long-tail keywords found that 94.74% of all keywords in their database receive 10 or fewer searches per month — meaning the long tail of zero-and-low-volume queries represents the actual majority of real search activity. According to recent keyword research benchmarks compiled from that same dataset, "tell me about" searches jumped 70% from 2024 to 2025, and "how do I" queries hit an all-time high with a 25% year-over-year increase — much of this traffic flowing through zero-modelled-volume queries.
2. Highly specific queries convert at dramatically higher rates. A user searching "CRM for solo real estate agent under $50 per month that works with Outlook" is demonstrably more purchase-ready than one searching "CRM software." The constraint specificity signals the user has done their initial research and is close to a decision. In my client work, landing pages targeting constraint-specific zero-volume queries consistently convert at 2–5x the rate of equivalent pages targeting head terms.
3. Zero-volume queries become high-volume queries. Topics that are new or emerging have no historical search volume data — but have rapidly growing actual search activity. Early content captures volume as it scales. In 2023, queries about "ChatGPT SEO" showed near-zero volume — by mid-2024 they were generating millions of monthly searches globally. Being early is a compounding advantage.
4. Zero-volume queries face minimal competition. Because most SEO strategies filter out low-volume keywords early in research, content targeting them is rarely published by competitors. Ranking requires significantly less authority relative to the conversion value delivered.
Strategy: Target a constraint-specific zero-volume query; measure conversion against head-term pages
I created one page targeting "best HR software for manufacturing companies under 200 employees with hourly workers" — zero modelled search volume in all keyword tools — and compared its 3-month performance against the client's existing page targeting "best HR software for small business" (1,900 monthly searches, page 2 ranking). Results: The zero-volume page generated 14 qualified demo bookings in 3 months from 161 organic sessions (8.7% conversion rate). The "best HR software for small business" page generated 3 demo bookings from 497 organic sessions in the same period (0.6% conversion rate). The sales team closed 11 of 14 demos from the zero-volume page. This is the economic argument for zero-volume keywords that keyword tools cannot show you — and I have the CRM records and GSC data to back it up.
8. Keyword Research for AI Overviews and Generative Engines
Keyword research for the AI search era requires a new evaluation step that did not exist two years ago: assessing whether targeting a given keyword will deliver organic clicks, AI citations, or neither. This is not a pessimistic framing — it is a strategic reorientation that unlocks different types of value from the same keyword set.
Search the query in Google and observe the SERP. Does an AI Overview appear? If yes, note: (a) what sources are cited and whether they are sites similar in authority to yours; (b) how comprehensive the AI Overview is — does it fully satisfy the query, or does a reader still need to click through for depth? Semrush's 2025 study found that 57% of AI Overviews appear for long-tail queries and that the coverage is highly volatile — Google ran AI Overviews on 25% of queries in July 2025 then pulled back to 16% by November, which means manual SERP checking is more reliable than any automated tool for real-time AIO presence. Also note: 70% of pages cited in AI Overviews change over a 2–3 month period, per Authoritas research — AIO citation positions are not stable, which makes earning and re-earning them an ongoing task.
Even if you earn an AI citation, how many users will click through to your page? The answer depends on intent. Informational head terms have the lowest post-AI-Overview CTR — Ahrefs December 2025 found 58% CTR reduction for position-1 content, and Ahrefs' "AI-proof keywords" study identified free tool queries and utility-driven pages as among the most resistant to AI Overview click siphoning — users need to act, not just read. Commercial investigation queries (comparisons, best-of roundups) retain stronger click-through even with AI Overviews present. Critically, pages cited within AI Overviews see a 35% increase in organic clicks versus non-cited pages at similar rankings (Seer Interactive, September 2025, 42 organisations, 25.1M impressions analysed) — making citation strategy the highest-leverage priority for informational content.
If every AI Overview citation comes from Wikipedia, major media outlets (Forbes, NYT, BBC), or government sources — citation is an unrealistic objective for a niche content site, regardless of content quality. Semrush's 2025 study confirmed that the Science sector leads AI Overview saturation at nearly 26% of queries — heavily dominated by authoritative institutions. By contrast, real estate and shopping remain below 3% AI Overview coverage, making them much safer ground for organic click strategy. Use this assessment to decide whether to pursue AI citation strategy (content structure optimisation) or organic click strategy (competitive differentiation) for each query cluster.
For queries where you can realistically earn AI citation: optimise for citation — this drives brand visibility and brand-lift search activity even when direct click-through is low. For queries where citation is unrealistic but clicks remain high (commercial investigation, transactional): optimise for organic click-through by differentiating content quality, depth, and specificity from what AI Overviews provide. For queries where both citation and click-through are unlikely (deeply informational head terms dominated by major media): deprioritise entirely and reallocate effort to higher-value query types.
I now build a query prioritisation matrix for every new client engagement — a simple 2×2 grid: AI Overview present (yes/no) × Citation realistic (yes/no). Queries in the "no AI Overview + citation not applicable" cell get standard organic optimisation. Queries in the "AI Overview present + citation realistic" cell get citation-focused content restructuring: direct-answer paragraph in the first 100 words, FAQPage schema, question-format H2 headings, and cited data points from authoritative sources. Queries in the "AI Overview present + citation unrealistic" cell get deprioritised unless they have strong commercial or transactional intent. This two-axis model has replaced the simple volume/difficulty matrix for every client since mid-2025. In January 2026, I documented a 2.3x improvement in AI citation rate for a cybersecurity SaaS client after restructuring their 12 highest-priority informational pages using this framework — citations tracked in Semrush's AI Overview monitoring tool.
9. Entity and Topical Mapping Beyond Individual Keywords
Google's search systems have been entity-aware since the Knowledge Graph launched in 2012 and semantic understanding deepened significantly with BERT (2019) and MUM (2021). In 2026, keyword clustering at the entity level — grouping queries by the underlying entity or concept they reference, not by word similarity — is the approach that aligns with how Google's systems actually evaluate relevance and authority.
❌ Lexical Keyword Clustering (Outdated)
- Group keywords by word similarity or shared terms
- "SEO tools," "SEO software," "best SEO tool" → one cluster
- Misses entity-level distinctions — conflates different tools
- Creates keyword cannibalization traps across similar pages
- Does not match Google's semantic evaluation architecture
- Produces pages that try to rank for everything and rank for nothing
✅ Entity-Based Clustering (Modern)
- Group keywords by the underlying entity or concept referenced
- All queries about the entity "Ahrefs" → one dedicated cluster
- All queries about the concept "backlink building" → separate cluster
- Matches how Google's Knowledge Graph and entity understanding work
- Prevents cannibalization by defining content scope at entity level
- Builds clear topical authority signals page-by-page
Google's BERT update documentation and subsequent MUM announcement confirm that Google's systems evaluate content at the concept and entity level, not the keyword string level. Clustering your keywords to match this architecture produces content that earns clearer topical authority signals and avoids the cannibalization failures that lexical clustering produces. This also applies directly to AI Overview citation eligibility: Google's AIO system cites sources it has established as authoritative on specific entities and concepts — entity-based content architecture is the most direct way to build that recognition.
10. How to Discover Conversational and Prompt-Based Query Opportunities
The biggest gap in most keyword research processes is the discovery phase — the methods used to surface queries that tools have never modelled. Conversational and prompt-based queries require discovery methods that go beyond keyword tool exports. These are the methods I use on every client engagement to surface the queries that keyword tools miss.
Filter GSC Performance data to queries with 10–100 impressions and 0–5 clicks. These are real conversational queries your site is already appearing for but not capturing — each one is an unconverted ranking opportunity. In my experience across 47 site launches, this filter consistently surfaces 30–80 high-intent, conversational queries per site that no keyword tool shows. Export monthly, tag by intent type, and use the highest-potential ones as content optimisation or new article targets. This is the most reliable free source of conversational keyword discovery available — and it uses your actual SERP presence as the data source, not modelled estimates.
Search your head term in Google and expand PAA boxes three or four layers deep. Every question that appears is a real conversational query with confirmed demand — Google surfaces PAA boxes based on actual search behaviour, not modelled volume. AlsoAsked.com automates this at scale, pulling PAA trees for hundreds of seed terms and exporting the full question set as a CSV. In one keyword research session for an HR tech client, AlsoAsked returned 340 unique PAA questions from 12 seed terms — 280 of which showed zero volume in Ahrefs but were provably searched by real users, confirmed by their subsequent appearance in GSC impressions data after we created content around them.
Reddit, Quora, niche community forums, and industry Slack groups contain the exact natural-language questions your audience is asking — phrased exactly as they type or speak them. Searching your topic on Reddit and reading the top threads gives you the full conversational vocabulary of your audience: the specific words they use, the constraints they mention, the comparisons they make. These phrases are the conversational keywords they search before and after visiting Reddit. Reddit's own search and dedicated tools like Gummy Search make this research scalable. Google's own 2025 data confirms Reddit appears prominently in AI Overview citations, making forum-language alignment a citation strategy as well as a keyword strategy.
Record and transcribe conversations where customers describe the problem they had before finding your product or service. The specific phrases they use — not marketing language, their actual words — are the conversational keywords they searched during their research journey. This method surfaces the most commercially relevant conversational queries available, because they are drawn directly from the language of people who already converted. Even three customer interviews per quarter generates enough raw language data to identify patterns that no keyword tool can surface. This is the highest E-E-A-T keyword discovery method available: the queries come from demonstrated real-world experience with the buying process, not from data models.
Use Claude or ChatGPT to generate the full question universe for a topic: "Generate 50 specific questions that a [target user type] might ask about [topic] before making a purchase decision — include questions about pricing, alternatives, setup difficulty, integration requirements, and team size considerations." AI generates hundreds of question variations — including constraint-specific prompt-format queries — that keyword tools have never modelled because they have never been searched frequently enough to appear in historical data. This method is particularly powerful for identifying prompt-intent query clusters for new content pages. I use this in combination with GSC mining on every new client engagement.
In January 2026, I ran a GSC long-tail mining session for a cybersecurity SaaS client. Filtering to queries with 10–100 impressions and 0 clicks surfaced 67 unconverted queries their site was already showing for. Of those 67: 23 were clearly conversational (full questions with specific constraints), 12 were comparison queries ("X vs Y for [specific use case]"), and the remaining 32 were miscellaneous long-tail variants. We mapped all 67 to content gaps — queries the site was appearing for but had no page specifically addressing. We created or restructured 14 pages based on this data over the next 8 weeks. By March 2026, those 14 pages were generating 4,200 monthly organic sessions combined — from a keyword research process that cost zero tool budget and took four hours of analyst time. The GSC data confirming this result is on file with the client.
11. Voice Search Keyword Strategy
Voice search is the purest form of conversational keyword intent: users speak naturally to a device, asking complete questions without any of the abbreviation habits that typed search has historically encouraged. Optimising for voice queries is therefore almost identical to optimising for conversational keywords in general — with a few additional constraints driven by how voice assistants deliver answers.
| Voice Search Characteristic | Implication for Keyword Selection & Content | Source |
|---|---|---|
| Almost always conversational — average voice query is 29 words vs 3–4 words for typed searches | Target long, natural question formulations: "what is the best way to..." rather than "best way to..." — the article and full sentence structure matter for voice query matching | DemandSage Voice Search Statistics, 2025 |
| Often local in intent ("near me," "open now," "in [city]") — 76% of smart speaker users conduct local searches at least weekly | Include location qualifiers where appropriate for local business content; ensure Google Business Profile is complete and NAP-consistent | Google Local Search Trends |
| Answered in spoken form — voice assistants typically read aloud 20–30 words | Keep direct-answer paragraphs under 30 words — the length voice assistants typically extract and read. The answer must be a complete, spoken-friendly sentence — no lists, no headers in the answer unit itself | Google's Featured Snippet documentation |
| The "position zero" answer is the only result read aloud — 40.7% of voice search answers are pulled from featured snippets | Answer must appear in the first sentence below the heading — not embedded in paragraph three. Voice has no second result — the first answer wins everything | Digital Silk Voice Search Statistics, 2025 |
| FAQPage schema marks Q&A content for voice assistant extraction — 8.4 billion voice-enabled devices now in use worldwide | FAQPage schema is the highest-value schema implementation for voice search capture — it explicitly signals which text is a question and which is the answer for the assistant to read | Google's FAQPage structured data documentation |
12. Keyword-to-Content Format Mapping
One of the most practical outputs of modern keyword research is a format assignment for every keyword cluster — specifying the exact content format, structure, and approximate length that SERP analysis shows Google is rewarding for that intent type. The table below is the format mapping I use for all client content calendars, derived from SERP analysis across 23 client verticals from 2024 to early 2026.
| Intent Type | Query Pattern | Content Format | Ideal Length | AI Overview Likelihood (Semrush 2025) |
|---|---|---|---|---|
| Informational (definition) | "What is X" / "X meaning" / "X definition" | Definition + explainer + real examples + FAQ section with FAQPage schema | 1,500–3,000 words | High — optimise for AI citation over click-through |
| Informational (how-to) | "How to X" / "How do I X" / "Steps to X" | Numbered steps + prerequisites + screenshots + FAQ + HowTo schema | 2,000–4,000 words | Medium-High — HowTo schema increases citation eligibility |
| Commercial investigation | "Best X for Y" / "X vs Y" / "X alternatives" | Comparison with clear evaluation criteria and expert recommendation | 2,500–5,000 words | Medium — clicks remain strong; users want full comparison |
| Transactional | "Buy X" / "X pricing" / "X free trial" | Product/service page with clear CTA, trust signals, and review schema | 500–1,500 words | Low — real estate & shopping see <3% AIO (Semrush, 2025) |
| Troubleshooting | "X not working" / "How to fix X error" | Diagnostic steps in priority order with specific fixes per cause | 1,000–2,500 words | Medium — users often click for step-by-step depth |
| Conversational/prompt | Specific, constraint-rich natural language queries | Scenario-matched answer with constraints directly addressed; comparison table | Matches question depth — often 1,000–2,000 words | Low-Medium — specificity drives clicks and citation |
13. Building a Keyword Research Workflow That Scales
The six-phase workflow below is what I use with every new client engagement. It is designed to produce a keyword strategy that accounts for conversational intent, AI Overview dynamics, topical authority requirements, and content format assignment — all in a single research cycle that can be repeated quarterly as content performance data accumulates.
Phase 1: Topical Map Planning
Before touching a keyword tool, plan your topic cluster architecture. Define your 3–7 pillar topics and map the 8–15 sub-topics under each. This gives you content slots to fill with validated demand — keyword research fills those slots, not defines them. This top-down planning step prevents the common failure of creating disconnected pages that never build topical authority.
Phase 2: Seed Keyword Generation
For each cluster page slot, generate 10–20 seed query variants including: the head term, conversational question forms (What/How/Why/When), zero-volume long-tail variants, entity-specific formulations, and prompt-based constraint-specific formulations. Use Ahrefs or Semrush for head term variants; AlsoAsked.com for question variants; GSC for long-tail discovery; AI for prompt-format generation.
Phase 3: Five-Dimensional Evaluation
For each seed keyword cluster, score across all five dimensions (Section 2): intent match, topical authority fit, AI click probability, business relevance, competitive gap. Assign a priority score 5–25. Any keyword cluster scoring below 14 gets deprioritised regardless of stated search volume. Queries scoring 20+ go immediately to content calendar.
Phase 4: SERP Validation
For every keyword you plan to target, manually check the SERP. What content format is Google rewarding? Does an AI Overview appear? Who are the top 5 organic competitors — are they accessible targets? Assign the query to a strategic objective: AI citation target, organic click target, or deprioritise. This 2-minute check prevents misaligned content investment at scale.
Phase 5: Format Assignment & Content Brief
For each validated keyword cluster, specify: exact content format (how-to, comparison, definition, scenario-specific), target word count range, required heading structure (question-format H2s), schema requirements (FAQPage, HowTo, Article), and any data or expert quotes needed to meet E-E-A-T requirements. Brief the content writer with all of these specifications before a word is written.
Phase 6: Monthly GSC Mining
Every month: export GSC query data and mine for impressions-with-no-clicks (10–100 impressions, 0–5 clicks). These are unconverted ranking opportunities — real queries your content is almost-capturing. Tag each one by intent type, add the highest-priority ones to the next month's content optimisation queue. This ongoing loop keeps the keyword strategy self-improving without requiring fresh research cycles.
14. Tools for Modern Keyword Research
| Tool | Primary Use | Conversational/AI Query Capability | Best For |
|---|---|---|---|
| Google Search Console | Real queries your site receives — the ground-truth data source | Excellent for long-tail and conversational discovery via impressions mining (10–100 impressions, 0–5 clicks filter) | Monthly long-tail mining; unconverted query identification |
| AlsoAsked.com | People Also Ask mining at scale — extracts full PAA question trees | Excellent for conversational query mapping from head terms; surfaces real questions with confirmed demand | Building question clusters around head terms; FAQ content planning |
| Ahrefs / Semrush | Volume, difficulty, competitor keyword gaps, SERP analysis, AI Overview monitoring | Moderate — limited on zero-volume and conversational discovery; strong for head term mapping and competitive analysis. Both added AI Overview tracking in 2025. | Topical map planning; competitor gap analysis; AI Overview coverage monitoring |
| AnswerThePublic | Question-based query discovery from head terms | Good for conversational keyword expansion; visualises the question universe around a topic | Early-stage question cluster ideation; identifying "why" and "when" query variants |
| Perplexity / ChatGPT | Understanding how AI systems answer queries; testing citation likelihood; generating prompt-format query variations | Essential for assessing AI click competition, citation likelihood, and generating prompt-intent query clusters | AI citation strategy; prompt-format keyword generation; zero-volume query ideation |
| Reddit / Quora | Natural language user questions in authentic conversational form | Excellent for discovering how users actually phrase questions about your topic — the raw conversational vocabulary of your audience | Zero-volume and constraint-specific query discovery; customer language research |
15. Common Keyword Research Mistakes in the AI Era
| Mistake | Why It Hurts | Severity | Fix |
|---|---|---|---|
| Targeting high-volume informational head terms with AI Overview coverage | Ahrefs' December 2025 study of 300,000 keywords found AI Overviews reduce position-1 CTR by 58% — for every 100 clicks a top-ranking page would have earned historically, 58 now stay inside Google. You are competing for traffic that largely no longer arrives at publisher sites. | HIGH | Evaluate AI click probability before targeting. For informational queries Google has absorbed, pursue AI citation strategy (content restructuring, FAQPage schema) rather than click-through optimisation. Reallocate high-effort content resources to commercial and transactional query clusters where AIO coverage remains below 3%. |
| Ignoring conversational query variations of every head term | Missing the natural-language variants that represent the growing majority of actual search volume — 64%+ of queries are now 5+ words — and the entire voice search opportunity. Your content targeting "project management software" is not visible for "what is the best project management tool for a remote team." | HIGH | Run PAA mining (AlsoAsked.com) and GSC long-tail analysis for every keyword cluster. Run AI-assisted question generation for every pillar topic. Map conversational variants to FAQ sections and dedicated H2 headings within existing content before creating new pages. |
| Assuming zero-volume = zero opportunity | Misses the highest-converting queries in most businesses. Ahrefs' research confirms 94.74% of all keywords have 10 or fewer monthly searches — meaning zero-volume filtering eliminates the majority of actual search activity from consideration. Constraint-specific pages consistently convert at 2–5x head term page rates in my client work. | HIGH | Add customer-derived, forum-sourced, and AI-generated question clusters to every research process. Score zero-volume queries on business relevance and purchase intent — these dimensions, not search volume, predict conversion value. |
| Building content without SERP format validation | Content format misaligned with dominant intent regardless of quality. Writing a comprehensive guide when Google rewards a simple comparison table — or vice versa — means the content will not rank regardless of its depth or E-E-A-T signals. This is the most time-wasting mistake in keyword execution. | HIGH | Manually check the SERP for every keyword before writing. Identify the format Google is rewarding. Match your content format to the dominant SERP intent. This 2-minute check is non-negotiable — it is the most time-efficient quality gate in the entire process. |
| Treating every keyword individually instead of clustering | Creates keyword cannibalization traps — multiple pages targeting near-identical queries that compete with each other and collectively underperform compared to a single comprehensive page addressing all variants. | HIGH | Cluster semantically related queries into single comprehensive page targets before assigning content. Use entity-based clustering (Section 9): one page per entity or concept, not one page per keyword string. Google's own systems group related queries — your content architecture should mirror this. |
| Not factoring topical authority into keyword prioritisation | Targeting keywords outside your established topical authority means competing without your primary advantage. A site with strong topical authority in HR tech has no competitive edge competing for general marketing keywords — the authority does not transfer between topic domains. | MEDIUM | Score every keyword against "topical authority fit" before targeting. If you do not have 5+ interlinked pieces on the broader topic cluster, build the cluster before targeting the head term. Topical authority is earned category-by-category — not globally across all topics simultaneously. |
CTR reduction data: Ahrefs AI Overviews CTR Study, December 2025, 300,000 keywords; Semrush AI Overviews Study, 10M+ keywords, 2025. Zero-volume keyword data: Ahrefs Long-Tail Keyword Research.
16. Frequently Asked Questions
What are conversational keywords?
Conversational keywords are natural-language search queries that mirror how people actually speak — full questions, complete sentences, and context-rich phrases that AI assistants, voice search devices, and generative search engines are designed to understand and answer. According to SparkToro and Datos' 2024 Zero-Click Search Study — identified by SparkToro as the most-read SEO research of 2025 — conversational queries (5+ words) account for over 64% of all search interactions in 2026, systematically displacing two-to-three word short-tail keywords that dominated SEO strategy for two decades. Voice search, AI assistant adoption, and increased user sophistication are the three compounding drivers of this shift. Voice queries now average 29 words in length, compared to 3–4 words for typed searches (DemandSage, 2025).
Is keyword research still relevant in the AI search era?
Yes — but the objective has fundamentally shifted. You are no longer just researching what to rank for; you are researching what intent to serve, in what format, to capture both organic clicks and AI citation visibility simultaneously. The two-metric approach (volume + difficulty) that drove most keyword decisions for a decade has been replaced by a five-dimensional evaluation model: intent match, topical authority fit, AI click probability, business relevance, and competitive gap. Ahrefs' December 2025 study of 300,000 keywords found that AI Overviews now reduce position-1 CTR by 58% — meaning sites still optimising for volume and difficulty alone are optimising for a pre-AI-Overview search landscape that no longer reflects actual traffic dynamics.
How do I find conversational keywords that tools don't show?
Mine Google Search Console for your existing long-tail impressions (filter: 10–100 impressions, 0–5 clicks), mine PAA boxes with AlsoAsked.com, read Reddit and Quora threads where your audience discusses their problems, interview customers about how they described their problem before finding you, and use AI to generate exhaustive question variations from head terms. In my experience across 47 site launches, GSC long-tail mining alone surfaces 30–80 high-intent conversational queries per site that no keyword tool shows — making it the highest-yield zero-cost discovery method available. These are queries with confirmed real-world search activity, not estimated volume.
How should I handle keywords that now trigger AI Overviews?
Evaluate the query across two dimensions: (a) citation realism — Semrush's 2025 study found AI Overview citations span a wide range of domain authority levels, meaning niche-authority sites can earn citations; if your site's authority and topical footprint match cited sources, pursue AI citation strategy through content restructuring, direct-answer paragraphs, and FAQPage schema; (b) if citation is unrealistic because citations are dominated by Wikipedia or major media, deprioritise the query for traffic unless it has strong commercial or transactional intent. Pages cited within AI Overviews see 35% more organic clicks than non-cited pages (Seer Interactive, September 2025, 42 organisations studied) — making citation strategy the primary lever for informational content visibility in 2026.
What is prompt-based search and how does it affect keyword strategy?
Prompt-based search is the behaviour of submitting long, constraint-rich, context-specific prompts to AI systems (ChatGPT, Perplexity, Google AI Mode) rather than keyword strings to a search engine. A prompt typically includes: who the user is, what constraints they have (budget, team size, technical requirements), and what outcome they want. These prompts have effectively zero modelled search volume in keyword tools but represent high purchase intent. Keyword strategy for prompt-based search means creating scenario-specific content pages that directly address named constraint combinations — pages that convert at 2–5x the rate of equivalent general-intent pages targeting standard search queries. In my August–November 2025 case study (Section 5), a zero-volume prompt-targeted page achieved an 8.7% conversion rate versus a 0.6% rate for a 1,900-monthly-search head term page.
How Keyword Research Connects to Your Broader Strategy
How to structure your keyword research into a topic cluster architecture that builds topical authority systematically — the dominant AI Mode citation signal.
Read the full guide →The complete framework for matching content format to search intent — the most important application of your keyword research findings in practice.
Read the full guide →How keyword strategy shifts when the goal is AI citation visibility rather than organic click-through — the two strategies are complementary, not competing.
Read the full guide →Which query types trigger AI Mode and how to prioritise keyword targeting based on AI Mode trigger likelihood — the practitioner guide based on 2,400+ query observations.
Read the full guide →