Skip to content

How to Measure Information Gain With Firecrawl and Claude

Before you commission the eleventh article on a topic, check what the ranking pages and AI answers already say, and what yours would add.

On this page
  1. Pulling the Pages That Rank and Reading Every Claim
  2. Sorting the Claims Into a Ledger
  3. Checking What ChatGPT and Google AI Mode Already Give Away
  4. Writing the Verdict and Deciding What Your Company Can Add
  5. Frequently Asked Questions

Empact Newsletter

One email a month on what is working in AI search and SaaS go-to-market.

Check your email for a link to confirm your subscription.

You are already subscribed.

Check your email address and try again.

We could not save your subscription just now; please try again.

Most topics a SaaS company gets asked to write about already have a full first page of decent articles on Google, and an AI answer printed above them. The eleventh article costs what the first one did, a writer’s week and an expert’s hour. If it says what the ten already say, nobody has a reason to link to it, and no AI answer has a reason to cite it.

The careful check before commissioning one is a spreadsheet of every claim the ranking pages make. Similarweb’s director of SEO and AI search puts it at about 45 minutes a keyword,1 so most teams skim the top results instead. Firecrawl and Claude now do that reading in minutes, beside what ChatGPT and Google AI Mode tell a buyer for free. The output is a verdict of go, go narrower or do not write.

We ran the check on this article’s own subject before writing it, and Claude’s verdict was go narrower. The steps below are that run. You need two accounts and about an hour, plus one thing from whoever asked for the article:

This judges a topic someone has already asked for. It does not find topics, and it does not write the article.

Empact Partners writes content marketing for SaaS companies, so their articles say what buyers cannot get from the pages that rank or an AI answer. The first thing we ask of a topic is whether an answer already gives it away.

Pulling the Pages That Rank and Reading Every Claim

The first three steps put every ranking page in one folder and turn each into rows of claims. Nothing is judged yet.

Step 1: Connect Firecrawl and write down the topic

Firecrawl connects to Claude Code through its hosted server. Run this once in a terminal, then type /mcp inside Claude Code and sign in to your Firecrawl account when the browser opens:4

Terminal
claude mcp add --transport http firecrawl https://mcp.firecrawl.dev/v2/mcp-oauth

Then make an empty folder and run git init in it. Git keeps each step’s output, so a bad turn costs one step and not the run. Two files go in before Claude Code opens there. CLAUDE.md holds the rules Claude Code loads in that folder, and topic.md holds the query and the reader. Our headless run used an API key instead of the sign-in.

CLAUDE.md
# Gain ledger

This folder decides whether one topic is worth an article. The topic, the search query and the reader are in topic.md.

## Rules
- Use Firecrawl for every search and every page read. Save what it returns before you use it.
- Never write the article. What this folder produces is the ledger and the verdict.
- A claim is one statement a page makes and expects its reader to believe: a definition, a number, a recommendation, a step, a named framework or an example. Write each claim as one plain sentence of your own, and keep the page's own words beside it.
- Keep each claim's evidence: the page's own data, a source it cites (name it), an example, or nothing.
- Merge two claims only when a reader would learn nothing new from the second. Shared words are not enough.
- When you cannot tell whether two pages agree, mark the claim contested and say why.
- Commit after every step.
topic.md
Topic: information gain in SEO
Query: information gain seo
Reader: the head of content at a B2B SaaS company, deciding whether to commission an article on this topic

The rule that decides the result is what counts as a claim. Most gap tools compare headings, and two pages with the same headings can say different things under them. A claim is one statement the reader is asked to believe, with its evidence beside it, and that is the unit an AI answer either repeats or has no use for.

Step 2: Search the query and scrape the pages that rank

First prompt
Read topic.md. Search the query with Firecrawl and save the first 10 results to cohort.csv with the columns rank, url, title and kind (article, forum thread, video or tool page).

Then scrape every article in the cohort with Firecrawl, main content only, and save each one as pages/<rank>-<domain>.md. Skip forum threads and videos, but keep them in cohort.csv. Tell me which pages you saved, how many words each one has, and any page Firecrawl could not read. Commit.

Firecrawl’s search returned nine of the ten pages Google ranked for the query that morning, in a slightly different order. Its tenth was a YouTube video where Google had a second Reddit thread. Claude scraped seven articles in about a minute. It kept the Reddit thread and the video as rows, as the prompt said, and it also skipped On-Page.ai’s guide, which it filed as a tool page.

Two things are not problems. The saved files still carry some navigation and promo blocks, so the word counts run high, and Claude ignores that chrome in the next step. And Firecrawl does not read Reddit at all, so read a forum thread in your browser if it ranks. Ours turned out to matter later.

Step 3: Have Claude write down every claim on every page

Second prompt
Read every page in pages/ in full. List every claim each page makes, following CLAUDE.md, and save them all to claims.csv with the columns page, section, claim, kind (definition, number, recommendation, step, framework or example), evidence and quote.

Then tell me how many claims each page makes, and which claims appear on one page only. Commit.

Claude read the eight pages in seven and a half minutes and wrote down 351 claims, each with the page’s own words beside it. Before it reported back, it checked every quote against its page with a script. The count by page already says something about the cohort:

Scroll horizontally to see all columns when the table is wider than this page.

The page Google ranks first makes the fewest claims, and 8 of its 25 appear nowhere else (18 November 2026)
PageClaimsOn no other page
Semrush258
Ahrefs7535
Search Engine Land4627
Similarweb7845
Animalz318
On-Page.ai2816
Customer Impact2914
ML for SEO3918

“On no other page” is not the same as new. Most of those claims are an example, a quote or a tool tip that one writer chose and another did not. The next step separates them from the claims a page backs with something of its own.

Sorting the Claims Into a Ledger

Rows of claims still read like eight articles. The ledger turns them into one table you can judge a topic from.

Step 4: Build the ledger and write down what nobody answers

Third prompt
Build ledger.csv from claims.csv: one row per distinct claim, one column per page with Y where the page makes it, a notes column, and a status column:
- consensus: half the pages or more make it;
- contested: two pages or more say different things about it (put each side in notes);
- own: one page makes it and backs it with its own data or its own example;
- shared: everything else.

Then write absences.md: the questions the reader in topic.md would still have after reading every page, and the evidence no page offers. For each one, name the page that comes closest and say why it falls short. Tell me how many rows each status has. Commit.

Claude merged the 351 claims into 214 distinct ones in three minutes. Each status tells you something different about the topic.

Scroll horizontally to see all columns when the table is wider than this page.

214 distinct claims from the eight pages ranking for “information gain seo” (18 November 2026)
StatusRowsWhat it tells you
Consensus10What any article on the topic is expected to say. A reader needs it, and it is no reason to write.
Contested6Where the ranking pages disagree. A side argued with evidence is a real opening.
Own45What one page backs with its own data or example. This is the gain the cohort already holds.
Shared153One writer’s example or tip with nothing behind it, or a point two or three pages make. Not a fault.

The ten consensus rows are the subject as everyone tells it. They cover what information gain is, the Google patent behind the term, original data as the way to add it, and AI answers dropping sources that repeat each other. Five of the eight pages give the definition its own section near the top, and the next step shows why a new article can leave that section out.

Read the contested rows slowest, because a dispute argued with evidence is something no single page gives a reader. The largest dispute runs through every page in the cohort. Two pages treat information gain as part of how Google ranks pages, and six say Google has never confirmed it.2 The Reddit thread in the cohort argues it in practitioners’ words:

Tips from Reddit

Whether information gain is a ranking factor

I’d just call that SERP gap analysis. It can be a perfectly useful content process without needing information gain as the explanation.

r/SEO, u/Legitimate-Salary108, October 2026

SEO’s favourite experiment: change 6 things, wait 3 weeks, then confidently attribute the outcome to thing #4.

r/SEO, u/Legitimate-Salary108, October 2026

The absences file is where a topic shows whether it has room. Claude found 13 questions a head of content would still have after all eight pages. Four of them matter most to anyone selling to SaaS buyers.

  • Cost and return. No page says what an information-gain article cost to make, or what it brought in as leads or pipeline.
  • Failures. Every case on every page is a success, so no page says when the work is not worth doing.
  • An independent number. Ahrefs, Similarweb and On-Page.ai rest their figures on one study, run by On-Page.ai, which sells the checker.
  • A disinterested source. All eight pages come from companies that sell something tied to the topic. Semrush also owns Search Engine Land, as that page discloses.

Those four are where a new article could add something. Whether yours can is the question Step 6 puts to you.

Checking What ChatGPT and Google AI Mode Already Give Away

A ranking page competes with the other pages on the results. An article published now also competes with the answer a buyer reads before clicking anything, so the engines’ answers go into the ledger as two more columns.

Step 5: Save the answers and mark the ledger against them

Ask ChatGPT and Google AI Mode the question the way your buyer would, and save each answer as a file in an answers folder. We asked what information gain in SEO is, and how to measure an article’s information gain before writing it. We pulled ours through DataForSEO’s API, and copying each answer out of the chat window works the same.

Fourth prompt
answers/ holds what ChatGPT and Google AI Mode said when we asked them the two questions in answers/questions.md. Read every answer. Add two columns to ledger.csv, chatgpt and ai_mode, and mark Y where that engine's answers already give the reader the claim in that row.

Then tell me which consensus rows the engines already give, which contested rows they settle one way without saying there is a dispute, and whether anything in absences.md is answered by an engine. Commit.

Google AI Mode already gave all ten consensus rows and ChatGPT seven of them. Neither engine gave any of the 45 rows a page backs with its own evidence.5 Both open their first answer on the definition, and Google prints the same definition in the AI Overview above the results.

The claims an AI answer leaves out are the ones a page backs with its own data or example.

Split by page, each article gets two bars: how much of it an AI answer already gives, and how much it backs with its own evidence. The pages built from shared and consensus claims are the ones an answer already reproduces:

How Much of Each Ranking Page an AI Answer Already Gives

The eight articles ranking for “information gain seo”, 18 November 2026

Already in an AI answerBacked by the page’s own evidence
View the data
LabelAlready in an AI answerBacked by the page’s own evidence
Animalz72%0%
Semrush58%5%
Customer Impact55%7%
ML for SEO44%0%
Ahrefs38%14%
On-Page.ai32%4%
Search Engine Land28%52%
Similarweb27%17%
Each page gets two bars: how much of it an AI answer already gives, and how much it backs with its own evidence.Source: Empact Partners, gain ledger of the eight articles ranking for

The page with no evidence of its own has nearly three-quarters of its claims in an answer. The one built on interviews behind its case studies is the page the answers leave mostly alone. If your draft would sit at the top of this chart, a buyer can get most of it without visiting your site.

The engines also settle two disputes without saying they exist. Both tell the reader to study the top results before writing, though one ranking page praises a writer who never looks at them. Each names its own number of pages to read, while the pages themselves argue for anything from three to twenty.

The second question was this page’s own subject. ChatGPT answered it with a competitor spreadsheet, a 100-point score and a ready-made prompt. Google AI Mode suggested a manual matrix of the top results, and feeding the scraped text of the top three into a language model. Neither told the reader to check the engines’ own answers, and neither ended in a decision.

Writing the Verdict and Deciding What Your Company Can Add

The ledger and the answers show where the room is. The verdict says whether anyone should pay to fill it.

Step 6: Ask for a verdict, with what the article would have to bring

Fifth prompt
Write verdict.md. Choose one: go, go narrower, or do not write.
- Go: an article can bring something the pages and the engine answers do not already give.
- Go narrower: only part of the topic has room. Name that part as the angle.
- Do not write: the pages and the answers already give the reader in topic.md what they need.

Give the reason in three sentences or fewer, citing rows of ledger.csv and items in absences.md. Then list what the article would have to bring to earn its place (a case, a first-party number, an expert's contrary view, a worked method), and mark each item "public" if you could produce it from public pages, or "ours" if only a person at our company could supply it. Commit.

Claude’s verdict was go narrower, toward “a measured, costed information-gain pass on B2B SaaS commercial pages (what it cost, what moved, what it returned in pipeline). Not an explainer on information gain.” Its reason was in the ledger. The consensus was already in the answers, and no answer touched the absences. It also named what that article would have to bring:

Scroll horizontally to see all columns when the table is wider than this page.

What Claude said an article on information gain would need. If nobody at the company can supply any of the first four, its verdict turns to do not write.
What the article would needWho could supply it
A before-and-after on B2B SaaS pages, with the baseline statedOurs
Cost per article, and the leads or pipeline the pages brought inOurs
Time from publishing to the first movement, with datesOurs
A practitioner who saw no lift, or a loss, and whyOurs
A small-data method with its sample size, cost and resultOurs
Gain scores for the pages ranking on commercial SaaS queriesPublic
A test of the pre-write measures against pages that already rankPublic
A check of Google’s documentation for any mention of the termPublic

Read the “ours” rows as a commissioning question. Each one points at a person, whoever holds the analytics, the costs, the customer or the contrary view. The public rows on their own make a thin article, which is why Claude put a condition on its own verdict.

That verdict asks for a before-and-after on a SaaS company’s own pages, with its costs and pipeline, and none of that comes out of a run like this one. So this page narrowed further, to what the run produced that no page and no answer had. That is the ledger, its numbers and the method that made them. Every ledger splits the same way, and the split is the point of running one:

On the left, what the run supplies: the claims on the ranking pages, their agreements and disputes, what AI answers give away and the open questions. On the right, what the company brings.
The run fills the left side for any topic in minutes. An article earns its place when someone at the company can fill the right side.

Claude filled the left side for this topic in about 14 minutes. Nothing on the right can come from a model, because a model only has what is already published. The right side is where a commission gets decided:

The verdict is the part I would never hand to a model. Claude can tell you what eight pages agree on and which of them has evidence of its own. It cannot tell you whether your head of product holds a number nobody has published, or whether a customer will let you publish theirs. That is the question a commission should turn on. When the answer is no, the right article is the narrower one you can back, or none at all.

Shanal GovenderSenior GTM Consultant @ Empact PartnersShanal Govender on LinkedIn

Empact Partners would start a topic like this from its absences, because that is the only place a new article can add something. Our first job is finding who inside your company can answer them. We learn the product in depth, interview the people who hold the case or the number, and write the piece with your team. A named consultant chooses the angle and signs the article off.

When the verdict is go, the next job is writing the article so it adds what the answers lack, and checking afterwards that ChatGPT and Google AI Mode quote it correctly. The playbook below covers both:

Once the article exists, run the same ledger on your own draft before it ships, with the draft saved as a ninth page. Every claim of yours that lands in the consensus column is a paragraph an answer already gives away, and the definition near the top is usually the first of them. What lands alone, backed by something only your company has, is the article.

For an article that is already live, the free check below reads it beside up to eight pages that rank for its search. It emails back what your article says that they do not, what it repeats, and what it is missing.

Keep the folder after the verdict. The pages that rank and the answers above them both change. Running the prompts again takes a quarter of an hour, so the next brief starts from what the topic looks like that week.

Frequently Asked Questions

Is information gain a Google ranking factor?

Nobody outside Google can say. The term comes from a Google patent filed in 2018 and granted in 2022, and Google has never said it uses it to rank pages.2 Treat it as a test of whether a page deserves to exist. The practical case holds either way. A page that repeats the ten above it gives nobody a reason to link to it, and gives an AI answer nothing it could not take from those ten.

How many pages should go into the ledger?

The first page of results, minus forum threads and videos, which leaves seven to nine articles on most topics. With fewer than five, the consensus column means little, because two pages agreeing can be a coincidence. Past ten, the extra pages mostly repeat the first ten, and the claims file grows past what you can read in an hour.

What does one ledger cost to run?

On Firecrawl, about ten credits: two for the search and one for each page.3 The free plan’s 1,000 monthly credits cover dozens of topics, and you can copy the engine answers out of the chat window for free. Claude Code needs a paid Claude plan, and our six prompts took about 14 minutes of Claude’s time. Your own hour goes on reading the ledger, which is the part worth paying for.

Can we run it without Claude Code?

Yes. Claude’s desktop app connects to the same Firecrawl server through its connectors, and the prompts work there as written. What you lose is the folder: no saved ledger to compare against next quarter, and every step depends on someone copying the last answer into the next prompt. For one topic that is fine. For a content plan of thirty topics it is a week of copying.

What if the verdict says do not write, and the topic is already on the plan?

Take the verdict to whoever put it there, with the ledger open. A do-not-write rarely means the subject is dead. It means the article as briefed would repeat the pages that rank. The absences file usually holds a narrower question someone at your company can answer, and that question is the brief worth commissioning instead.

How is this different from a content gap analysis or a content score?

A gap analysis compares keywords and headings, so it finds subtopics your page has not covered yet. A content score rewards a draft for covering what the ranking pages cover. The ledger works at the level of claims and asks the opposite question: what would this article say that no ranking page and no AI answer says already. A draft can score well and add nothing.

Why check ChatGPT and Google AI Mode if our buyers search on Google?

Because Google answers first now. For “information gain seo”, Google’s AI Overview printed the definition, the patent and the manual audit above the first result on the day we ran this. An article whose main points sit in that summary competes with a free answer placed above it. ChatGPT and AI Mode show the same thing from the other side: what a buyer learns without opening any page.

What counts as information gain for a SaaS company with no research budget?

What your company already knows and nobody outside it can copy. What your support team hears every week, what your product data shows about how customers use a feature, a customer story with its numbers, the call your sales lead makes that the market gets wrong. None of it needs a survey. It needs someone to ask the person who knows, before the brief is written.

Should we unpublish old articles the ledger shows are pure consensus?

Not on the ledger alone. Check the page’s own clicks and links first. A consensus page that still brings visits and leads gets refreshed with something only you have, and a second ledger run shows whether the new version holds anything the answers do not. One that brings neither joins the posts a blog should cut or merge.

How far can we trust Claude’s ledger?

Trust the quotes and check the judgment. In our run Claude tested every quote against its page with a script, so the words are the pages’ own. Merging two claims, and calling a row consensus or contested, is judgment. Read the contested rows and the absences yourself before the verdict. Ours filed a vendor’s guide as a tool page until we said otherwise.

How often should we run the ledger again?

Before each brief, and once more before the article ships, because both sides of the comparison move. The ranking pages for a busy topic change within months, and the AI answers change with the pages they cite. A ledger from last quarter shows where the gap was three months ago.

Which topics should we run the ledger on first?

The ones already commissioned and not yet written, starting with any brief that opens on a definition. Those are the pieces most likely to repeat what ranks. After that, take the questions your sales team hears on calls, because a buyer’s question no ranking page answers is the likeliest go verdict on your plan.

Sources (5)
  1. Limor Barenholtz, Similarweb, "Information Gain: How To Measure, Score, And Use It For AI Search" (2026)↩124 June 2026.
  2. Google LLC, "Contextual estimation of link information gain" (2018)↩1↩2US patent 11,354,342, filed 2018, granted 7 June 2022.
  3. Firecrawl, "Pricing" (2026)↩1read 18 November 2026.
  4. Firecrawl documentation, "Claude Code quickstart" (2026)↩1read 18 November 2026.
  5. Empact Partners, gain ledger of the eight articles ranking for, "information gain seo" (2026)Empact Partners' own data↩1↩2run on 18 November 2026: 351 claims scraped with Firecrawl and sorted by Claude Code, checked against ChatGPT’s and Google AI Mode’s answers to two questions.

Ready
To Connect?

Let's Partner