Search Intent Classification in Python: Read the SERP, Not the Keyword

I was cleaning up my keyword shortlist last week when I hit a keyword that broke my intent workflow: “ai seo tools”.

Every keyword-text heuristic I had said informational. No “buy”, no “pricing”, no brand name. The modifier “tools” reads like a research query — someone learning what exists. My keyword-based classifier — the one from my search intent guide, the one I’d been using happily for months — tagged it informational, and I queued up a 3,000-word explainer.

Then I looked at the actual SERP. Three listicles. Three review pages. Three “best of” articles. Not one definitional explainer in the top 10. Google was rewarding comparison content, not education. If I’d published the explainer, I’d have spent two weeks writing into a wall — the exact failure mode that motivated the SERP difficulty scorer, just one layer up.

That’s when I finished the tool I’m writing about today. serp_intent classifies intent from what Google does, not from what the keyword says — and cross-validates the two, because the disagreement is the signal.

The problem with reading keywords

Keyword-text intent classification is layer 1, and it’s genuinely useful: “how to” is informational, “buy X online” is transactional, a brand name plus “login” is navigational. It’s free, instant, and needs no API. I wrote a whole guide defending it.

But it fails in three specific, expensive ways:

Modifiers are niche-dependent. “Best” sounds commercial — until the SERP for “best bitcoin wallet” turns out half educational. “Tools” sounds informational — until “ai seo tools” turns out commercial. The same word means different things in different niches, and the keyword can’t tell you which.

Intent drifts. A keyword that was purely informational two years ago may be commercial today. Your keyword-text classifier uses the same static rules it always did; Google re-runs its experiment every single day.

Mixed intent is invisible. “Workflow automation” might be 60% informational / 40% commercial. A single-label classifier picks one and throws away the mix — the mix is exactly what determines your page structure.

The fix for all three is the same: look at the results. Google has already run the intent experiment on millions of real searchers. The page types it puts in the top 10 are the answer key. You just need to read it systematically — which is a classification problem, and classification problems are what Python is for.

The tool: one command, one API call

serp_intent ships in the zens-ink package (the same 17-tool CLI set covered in the tools overview and exposed to AI agents via the MCP server). Setup:

pip install zens-ink

# Serper.dev key in a .env next to where you run it
echo 'SERPER_API_KEY=your-key' > .env
# (free tier: 2,500 searches/month — https://serper.dev)

python3 -m zens_ink.serp_intent "best crm for small business"

That’s one API call per keyword. The output from a real run this week:

==================================================================
  SERP Intent Analysis: best crm for small business
==================================================================

  Dominant Intent: INFORMATIONAL

  Intent Scores (0-3):
    informational   █░░ 1.3
    commercial      █░░ 1.0
    transactional   ░░░ 0.0
    navigational    ░░░ 0.0

  SERP Page Types:
    blog            3 (30%) ■■■
    article         2 (20%) ■■
    comparison      2 (20%) ■■
    forum           1 (10%) ■
    listicle        1 (10%) ■
    video           1 (10%) ■

  SERP Features: people also ask

  ── Recommendation ──
  Format:     In-depth guide / tutorial (BLUF: answer first, then elaborate)
  Structure:  H2/H3 hierarchy, FAQ section, clear definitions up top
  Split:      Mixed intent SERP — 57% informational / 43% commercial.
              Structure page to satisfy both paths.
  AI Risk:    [HIGH]
  ...
  ⚡ Fragmented SERP (6 page types) — mixed signals, hard to satisfy all intents

  ── Top 10 Results ──
     1. [forum       ] www.reddit.com
        Best and inexpensive CRM for small business
     2. [article     ] www.zoho.com
        The Best CRM Software for Small Businesses
     ...

Three things in this output are worth more than the “dominant intent” label at the top.

First, the split: 57/43 informational/commercial. This SERP doesn’t have a single intent, and the tool refuses to pretend it does — it tells you to structure for both paths (answer-first up top for the informational half, comparison table for the commercial half).

Second, Reddit at #1. A forum thread outranking Zoho, PCMag, and the US Chamber of Commerce is Google explicitly saying it rewards first-hand experience for this query. That’s a content-strategy input no keyword-text classifier will ever give you.

Third, the fragmentation warning: six different page types in ten results. Fragmented SERPs are genuinely hard — the tool flags them instead of confidently averaging away the signal.

How the classification actually works

Under the hood it’s three signal layers, applied per result, deterministic and auditable — no ML model, no training data, no drift you can’t read in the source.

Layer 1 — domain overrides. Some domains have exactly one content type, regardless of URL path: wikipedia.org is always wiki, reddit.com always forum, youtube.com always video, amazon.com always shopping, github.com always repository. ~40 hardcoded domains short-circuit everything else.

Layer 2 — URL path patterns. Priority-ordered regex rules on the path: /pricing, /plans, /checkout → pricing; /calculator, /checker, /converter → tool; /vs/, /compare, /alternatives, /best-, /top- → comparison; /blog/, /guide/, /how-to, /tutorial → blog; and so on down to legal and about pages.

Layer 3 — title patterns. Titles carry signals paths can’t: a numbered title (^(\d+\s|top\s+\d+|...)) makes a page a listicle even on a flat URL; vs\.?|versus|alternatives to marks comparison; review|hands-on|tested marks review. Title beats path because it’s what the publisher claims the page is.

Each result gets exactly one of 20 page types: blog, article, wiki, documentation, guide, tool, listicle, comparison, review, directory, pricing, product, shopping, homepage, forum, video, social, legal, about, repository.

The scoring: position-weighted 0-3

Page types then map to intent weights, and this mapping is where the domain knowledge lives:

Page type	Informational	Commercial	Transactional	Navigational
blog / wiki / docs	3	—	—	—
listicle / comparison / review	—	3	—	—
pricing / shopping	—	1	3	—
product	—	1	2	—
tool (free calculator etc.)	1	—	2	—
homepage	—	—	—	1
forum	1	1	—	—
repository	2	—	1	—

Two mappings deserve comment because they encode real intent behavior. A tool page scores transactional-heavier than informational — a “convert pdf to word” query is a do query; the searcher wants to use something, not read about it. And a homepage only scores 1 on navigational, because a tool’s homepage ranking #1 is usually a navigational-brand signal but could just be a strong product — the tool deliberately under-commits.

Then comes position weighting. Each result’s weights are multiplied by a position weight decaying linearly from 1.00 at position 1 to 0.37 at position 10, summed per intent, and normalized to a 0-3 scale. The decay encodes something every SEO knows: Google signals intent most strongly through what it ranks first. A pricing page at #1 says more than a pricing page at #9.

You can see the decay matter in the cleanest run I captured — “buy bitcoin”:

  Dominant Intent: TRANSACTIONAL

  Intent Scores (0-3):
    informational   ░░░ 0.0
    commercial      █░░ 1.2
    transactional   ██░ 2.8
    navigational    ░░░ 0.0

  SERP Page Types:
    shopping        8 (80%) ■■■■■■■■
    listicle        1 (10%) ■

  AI Risk: [NONE]
           LOW — transactional queries rarely get AI Overviews.
           Focus on conversion.

Eight shopping pages, transactional 2.8, and — note this — AI risk LOW. That’s the other layer: the tool reads SERP features (AI Overview, PAA, featured snippet, shopping, knowledge graph) and pairs dominant intent with an AI-Overview exposure assessment. Transactional queries rarely trigger AI Overviews; informational queries do, constantly. If you’re writing into a HIGH-risk informational SERP, the differentiation requirement (original data, proprietary frameworks — the GEO playbook) isn’t optional. The intent layer decides what to build; the AI-risk layer decides how defensible it is.

Cross-validation: the flag that matters

The single most useful flag is --cross, which runs the keyword-text classifier first, then compares:

python3 -m zens_ink.serp_intent "ai seo tools" --cross

  Dominant Intent: COMMERCIAL

  Intent Scores (0-3):
    informational   ░░░ 0.3
    commercial      █░░ 1.8
    ...

  ⚠ Cross-validation: mismatch (confidence: medium)
    Keyword text suggests 'informational' but SERP shows 'commercial'.
    Trust the SERP — Google's behavior is ground truth.
    The keyword modifier may be misleading.

This is the exact keyword from the opening story, run through the tool that exists because of it. Keyword text: informational. SERP: listicles and reviews at 60%+, commercial 1.8. The tool doesn’t average the two or ask you to pick — it rules for the SERP, because Google’s ranking behavior is the outcome of a running experiment and the keyword text is a hypothesis.

The two-layer design isn’t redundancy; the two layers have opposite failure modes. Keyword-text classification is high-precision on unambiguous modifiers (“how to”, “buy”) and wrong on niche-dependent ones. SERP classification is always grounded in current reality but costs an API call and can mislabel individual pages. Agreement = high confidence, publish. Disagreement = the SERP wins, and you’ve just caught a mistake before making it. For ~200 keywords a month, this costs a fifth of the free Serper quota — the pricing trade is computed in the Serper vs Brave benchmark.

Where the two layers fit in a full pipeline: fan-out planning generates the sub-query candidates, clustering groups them, kd scores how hard each cluster’s SERP is, and serp_intent decides the format each cluster’s page takes. Difficulty tells you whether to fight; intent tells you what to bring.

The honest limits

One type per URL. A page can be a listicle and a comparison. The classifier picks one. On heavily mixed pages this loses information the distribution was supposed to capture.

“Article” is a weak signal. Generic article pages (no path signal, no title pattern) score informational with weight 1 — deliberately low confidence. SERPs dominated by them (see the “semrush login” run: 7 articles, 2 homepages) will read as mildly informational even when a human immediately sees a navigational brand query. The heuristic has no brand-entity model.

Homepage ambiguity. A brand homepage at #1 gets navigational 1 — but it might be a tool ranking on merit. Brand-heavy SERPs are the tool’s weakest case; cross-check domain concentration yourself.

Drift between runs. The SERP is live data. Run it twice a month apart and you can get different answers — which is correct behavior (intent drifts) but means you should re-check before committing weeks of writing, not rely on a snapshot forever.

One API call per keyword. Fine at shortlist scale (hundreds), wrong at corpus scale (tens of thousands). For corpus-scale intent labeling, layer 1 (keyword text) is still the right first pass — then spend SERP calls only where the ambiguity or the stakes justify it.

Try it

pip install zens-ink
python3 -m zens_ink.serp_intent "your keyword here" --cross
python3 -m zens_ink.serp_intent "生辰八字" --zh     # Chinese SERPs
python3 -m zens_ink.serp_intent "kw" --json         # for pipelines

The next keyword on your shortlist is probably classified wrong by its text. Takes one API call to find out which one.

FAQ

Why classify search intent from the SERP instead of from the keyword text?

Keyword-text classification relies on modifiers ('best', 'how to', 'buy') that are ambiguous across niches. 'Best X' is commercial in some niches and informational in others, and intent drifts over time as SERPs change. Classifying the actual top-10 results by page type reads Google's own answer to the intent question — the search engine already ran the experiment on millions of searchers, and the page types it rewards are the verdict. The two approaches are combined with a cross-validation flag: agreement gives high confidence, disagreement means trust the SERP.

How does the SERP intent classifier label each result, and how accurate is it?

Each organic URL is labeled with one of 20 page types using three signal layers applied in order: domain overrides (wikipedia.org is always wiki, reddit.com always forum, amazon.com always shopping), URL path regex rules (/pricing, /vs/, /calculator, /blog/), and title patterns (numbered listicles, 'vs' comparisons, review language). It is a heuristic, not machine learning — deterministic and auditable, but it assigns exactly one type per URL and treats generic 'article' pages as a low-confidence informational signal. On brand queries whose SERP is full of the brand's own sub-pages, scores can read oddly; treat the output as a structured first opinion, not ground truth.

What are the 0-3 intent scores computed from?

Every page type maps to intent weights (a listicle is commercial 3; a pricing page is transactional 3; a free tool page is informational 1 + transactional 2, because 'do' queries want to use something). Each result's weight is multiplied by a position weight that decays linearly from 1.00 at position 1 to 0.37 at position 10, summed per intent, and normalized to a 0-3 scale. The decay encodes that Google signals intent most strongly through what it ranks first, and the normalization keeps scores comparable across SERPs even when fewer than 10 results return.

How much does it cost to run, and what do I need?

One Serper.dev API call per keyword (the tool requests the top 10 organic results). Serper's free tier is 2,500 calls/month, so a 200-keyword intent audit costs 200 calls — under a tenth of the free quota. Setup is a Serper API key in a .env file and pip install zens-ink; the tool itself is pure Python standard library with zero dependencies. Add --zh for Chinese SERPs (gl=cn, hl=zh-CN) or --gl/--hl for any other locale.

How does this fit with keyword difficulty scoring?

They answer different questions and run in sequence. Keyword difficulty (the kd tool) tells you how hard the SERP is to enter; SERP intent tells you what content type Google will actually reward once you're competing there. A low-difficulty keyword with mismatched intent is still a dead end — you can't outrank a page type. The practical pipeline is: shortlist by difficulty, then run serp_intent on the shortlist to decide format (guide, tool, comparison table, landing page) before writing anything.