Referring Domains Without an API: Scanning the Common Crawl Graph Yourself

Every backlink API sells you the same deal: one number, rented, rate-limited, and gone when the subscription lapses. There is an alternative nobody mentions in tutorials: the Common Crawl foundation publishes the entire domain-level web graph as open data. Download it once and you have referring-domain counts, competitive comparisons, and PageRank for 133 million domains — offline, forever, for the cost of a disk.

This post is the pipeline I built last week. Real commands, real output, and the numbers it produced for ten sites, including two surprises I did not expect.

What you actually download

Common Crawl publishes a domain-level graph per quarterly crawl. The cc-main-2026-jul-aug-sep release:

• 133 million domains (vertices file, 976MB)
• 2.15 billion edges (8.6GB compressed) — one line per source-domain → target-domain relationship, deduplicated
• Host ranks (2.5GB) — precomputed PageRank per domain, so you get an authority score in the same download

Hostnames are in reverse-domain notation (com.example.subdomain), which trips everyone up the first time. Your scanner just reverses the labels before matching.

The files live on data.commoncrawl.org under projects/hyperlinkgraph/. No account, no key, no request cap. curl and patience are the only prerequisites.

The scan pipeline

The tool has three commands. First, fetch and prepare the graph (one-time, ~30 minutes on a domestic connection):

$ python3 cc_linkgraph.py init
Downloading cc-main-2026-jul-aug-sep vertices... 976MB done
Downloading edges... 8.6GB done
Loading vertices into SQLite... 133,080,311 domains
Building reverse-domain index... done

Then scan your targets — a plain text file, one domain per line:

$ python3 cc_linkgraph.py scan --targets targets.txt
[1/13] zens.ink                → 5 referring domains
[2/13] uzenlabs.com             → 5 referring domains
[3/13] theonchaindiary.com      → 2 referring domains
[4/13] rekt.news                → 635 referring domains
[5/13] slowmist.com             → 365 referring domains
...
Scan complete: 13 targets, 2.15B edges read, 19m 04s

One full pass over the edge list took 19 minutes on a laptop. Every result lands in a SQLite snapshot, so re-running report later is instant:

$ python3 cc_linkgraph.py report
target              referring_domains   top_competitor_overlap
rekt.news           635                 slowmist.com (41 shared)
slowmist.com        365                 rekt.news (41 shared)
zens.ink            5                   uzenlabs.com (4 shared)

Finding one: your own site cluster is an echo chamber

The scan produced an uncomfortable chart for my own portfolio. The brand hub showed five referring domains — and when I listed them, all five were my own sites. Footer links, JSON-LD sameAs references, GitHub org profiles linking back. Third-party referring domains: zero.

That is not a backlink profile. That is an echo chamber with extra steps. Ahrefs had been showing a DR in the thirties for the same domain, and the number felt fine — until the edge list made it obvious the authority was almost entirely self-referential. DR counts the links; it does not ask whose they are.

The fix this surfaced: stop interlinking harder, start earning one real third-party mention. A single link from a domain neither of my sites controls is worth more than twenty more cross-links inside the cluster.

Finding two: the graph is a floor, not the truth

My content site with an Ahrefs DR of 17 showed exactly 2 referring domains in the Common Crawl graph. Ahrefs sees dozens. Neither number is wrong:

• Common Crawl samples the web weighted toward high-PageRank pages. Links on pages the crawler never fetched do not exist in the graph.
• Ahrefs crawls aggressively for links but still samples. Its index for small sites is also incomplete — just incomplete differently.

The practical rule I landed on: use the graph for relative, competitor-scale comparisons, and use Google Search Console’s links export for your own sites’ ground truth. GSC’s data comes from Google itself — the only crawler whose opinion of your backlinks affects your rankings. The graph’s advantage is that it covers competitors GSC will never show you.

For competitive intel the floor is good enough: rekt.news at 635 referring domains and slowmist.com at 365 versus my 2 is not a gap that sampling error explains. The ceiling is the story, not the decimal.

Finding three: GSC exports + the graph = an outreach radar

The last piece combines two free sources. Google Search Console exports every domain that links to you; the graph tells you which pages on the web mention your competitors. The overlap you care about: domains that link to a competitor but not to you.

The pipeline that automates this:

1. Pull candidate pages ("best X", "alternatives to Y" roundups, curated directories) from search results
2. Fetch each page and verify it actually links to a named competitor — not just contains the words
3. Score and triage: page authority, topical fit, whether the page already links to you
4. Output a ranked outreach list, split into gold / nurture / review buckets

Step 2 matters more than it sounds. Roughly a third of “alternatives” roundup pages in my test set mentioned the competitor only in body text or an image caption — no actual link. Those are unlinked mentions, a different (and honestly easier) pitch. The verification step is what separates a link radar from a scrape-and-spam list.

When to just use the free API tier

Honesty section. The graph is the wrong tool when:

• You need one domain’s fresh data now. A free API lookup takes 30 seconds; a graph download takes an evening. CrawlGraph’s free tier (15 lookups/month) or Ahrefs’ free API both beat the graph for single-domain spot checks.
• Your site is new. The quarterly snapshot lags by definition. A link earned last week is not in it.
• You only track 2–3 domains. The graph’s advantage is bulk and repeatability across dozens or hundreds of domains. Below that threshold the setup cost dominates.

The graph is the right tool when you track a portfolio, want to diff quarter over quarter, or want numbers no API tier will give you at any price — like how many of your “referring domains” are just you.

The whole stack, priced

• Common Crawl domain graph: free, quarterly
• GSC links export: free, on demand
• The scanner + radar pipeline: ~300 lines of Python, stdlib only
• Total API budget: $0

That is the entire point. Not that paid tools are a scam — their indexes are deeper and fresher than any public dataset. But for indie sites, the marginal decision isn’t “Ahrefs vs free data.” It’s “any data vs no data,” and the public graph has been sitting there the whole time.

FAQ

Is the Common Crawl domain graph really free?

Yes. It's a public dataset on AWS S3 (and Common Crawl's own mirrors). You can download the vertices, edges, and host-ranks files without an account. The only cost is bandwidth and your own compute time — the full edge list is 8.6GB compressed.

Why does it show fewer referring domains than Ahrefs for my site?

The crawl that builds the graph is a sample of the web, weighted toward high-PageRank pages. For a small site with DR under 20, many of your links live on pages the crawl never fetched. Treat Common Crawl numbers as a floor and use Google Search Console's links export for your own sites' ground truth.

How often is the graph updated?

Quarterly. The domain-level graph for July–September 2026 (cc-main-2026-jul-aug-sep) shipped in October 2026. If you need fresher data for a single domain, a free API tier is more practical; the graph's advantage is bulk and competitor-scale analysis.