How to Log What ChatGPT and Claude Search Before They Answer

An AI answer shows you who got named. It does not show you what the engine looked up first, and an engine can only cite a page that came back from one of its own searches.

The steps below log those searches for your buyers’ questions, on ChatGPT and Claude, through one API, and keep the ones that come back run after run. They tell you which pages to write and which lists to get onto.

Empact Partners runs this log in our Generative Engine Optimization (GEO) work, the part of our go-to-market consultancy that gets B2B software brands named when buyers ask AI engines about their category. We ran it on our own category on 30 October 2026: eight buyer questions, five runs of each on both engines. What came back:

Before the first call, you need:

The procedure logs what the models search when they are called through their APIs. It does not record what a buyer sees in the ChatGPT app, how often buyers ask a question, or anything about Google.

We run it through DataForSEO’s LLM Responses API, which calls ChatGPT and Claude with web search switched on and returns both engines’ searches in one format, on one account.

The other route is a key with each vendor. OpenAI documents a web search record that “will usually (but not always)” include the queries, and Anthropic documents every query Claude ran, at $10 per 1,000 searches plus tokens. We have not run that route.

Log the Searches Behind Five Answers

Step 1: Write the questions file

Make a new folder, open a plain text editor, paste your buyer questions one to a line, and save the file in that folder as questions.txt. These are the eight we ran:

company that can make our startup show up in ChatGPT recommendations
top GEO consultancies for B2B SaaS
affordable AI search consultancy for startups
AI search consultancy for scaling from Series A to Series B
company that builds mentions and UGC so AI engines recommend our software
our SaaS never comes up in ChatGPT, which company can change that?
company that helps B2B brands get featured in AI Overviews
consultancy to get our product recommended when people ask AI for the best tools

The eight come from the Empact Panel, the measurement we built to track what ChatGPT, Google AI Mode, and Claude say about our partners’ categories every week. On our own category it tracks 500 questions, and we took eight whose honest answer is a list of companies, with no brand in them.

Keep the file the same for every run. The questions are the experiment, and a question edited halfway through is a different question.

Step 2: Save the script

In the same folder, save the script below as log_searches.py. We wrote it in Claude Code, and it is the exact file that produced every number on this page. To use other models, edit the two model names in ENGINES. To search from another country, change "US" in ask().

#!/usr/bin/env python3
"""Send buyer questions to ChatGPT and Claude through DataForSEO's LLM Responses API,
log every web search each model ran before answering, and rank the searches that recur.

  export DATAFORSEO_LOGIN=...  DATAFORSEO_PASSWORD=...
  python3 log_searches.py --pilot           # question 1, once on each engine
  python3 log_searches.py --repeats 5       # the full run; calls already bought are reused

Writes raw/ (every response), runs.csv (one row per call), searches.csv (one row per
search) and ranked.csv (each search, and in how many runs each engine made it).
"""
import argparse, base64, csv, json, os, re, urllib.error, urllib.request
from pathlib import Path

ENGINES = {  # engine: (DataForSEO path segment, model)
    "chatgpt": ("chat_gpt", "gpt-4.1-mini"),
    "claude": ("claude", "claude-haiku-4-5"),
}
API = "https://api.dataforseo.com/v3/ai_optimization/{}/llm_responses/live"


def ask(segment, model, question, auth):
    body = [{"user_prompt": question, "model_name": model, "web_search": True,
             "web_search_country_iso_code": "US", "max_output_tokens": 300}]
    req = urllib.request.Request(API.format(segment), data=json.dumps(body).encode(),
                                 headers={"Authorization": "Basic " + auth,
                                          "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=180) as r:
        return json.load(r)


def find(obj, key):
    """Every value stored under `key`, anywhere in the response."""
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == key:
                yield v
            yield from find(v, key)
    elif isinstance(obj, list):
        for v in obj:
            yield from find(v, key)


def read(resp):
    task = resp["tasks"][0]
    result = (task.get("result") or [{}])[0]
    flat = lambda key: [x for v in find(result, key) if v for x in v]
    return {"status": task["status_code"], "cost": task.get("cost") or 0,
            "money_spent": result.get("money_spent") or 0,
            "web_search": result.get("web_search"), "searches": flat("fan_out_queries"),
            "pages": len(flat("search_results")), "cited": len(flat("annotations"))}


def norm(s):
    return re.sub(r"\s+", " ", re.sub(r"[^\w\s]", " ", s.lower())).strip()


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--questions", default="questions.txt")
    p.add_argument("--repeats", type=int, default=5)
    p.add_argument("--pilot", action="store_true", help="question 1, once per engine")
    p.add_argument("--budget", type=float, default=3.00, help="never pass this, in USD")
    a = p.parse_args()

    login, password = os.environ["DATAFORSEO_LOGIN"], os.environ["DATAFORSEO_PASSWORD"]
    auth = base64.b64encode(f"{login}:{password}".encode()).decode()
    questions = [l.strip() for l in open(a.questions) if l.strip() and not l.startswith("#")]
    questions, repeats = (questions[:1], 1) if a.pilot else (questions, a.repeats)
    Path("raw").mkdir(exist_ok=True)

    rows, costs, spent, lost = [], [], 0.0, 0
    for qi, question in enumerate(questions, 1):
        for run in range(1, repeats + 1):
            for engine, (segment, model) in ENGINES.items():
                f = Path(f"raw/{engine}-q{qi:02d}-r{run}.json")
                if f.exists():  # bought already: read it, never pay twice
                    got = read(json.loads(f.read_text()))
                else:
                    nxt = max(costs) if costs else 0.10
                    if spent + nxt > a.budget:
                        print(f"Stopping before ${a.budget:.2f}: the next call could pass it.")
                        return finish(rows, spent, lost)
                    try:
                        resp = ask(segment, model, question, auth)
                    except urllib.error.HTTPError as e:  # refused, so not charged
                        print(f"Refused: HTTP {e.code} {e.read()[:200]!r}")
                        return finish(rows, spent, lost)
                    except Exception as e:  # may have been charged: never re-send it
                        print(f"q{qi} run {run} {engine}: no answer ({e}), not re-sent")
                        spent, lost = spent + nxt, lost + 1
                        continue
                    f.write_text(json.dumps(resp, indent=1))
                    got = read(resp)
                spent += got["cost"]
                costs.append(got["cost"])
                rows.append({"question_no": qi, "question": question, "run": run,
                             "engine": engine, "model": model, **got})
                print(f"q{qi} run {run} {engine:7}  searches {len(got['searches'])}  "
                      f"cited {got['cited']:2}  ${got['cost']:.4f}  total ${spent:.4f}")
    finish(rows, spent, lost)


def finish(rows, spent, lost):
    with open("runs.csv", "w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["question_no", "question", "run", "engine", "model", "status", "web_search",
                    "searches", "pages", "cited", "cost_usd", "money_spent_usd"])
        for r in rows:
            w.writerow([r["question_no"], r["question"], r["run"], r["engine"], r["model"],
                        r["status"], r["web_search"], len(r["searches"]), r["pages"],
                        r["cited"], r["cost"], r["money_spent"]])
    with open("searches.csv", "w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["question_no", "question", "run", "engine", "search"])
        for r in rows:
            for s in r["searches"]:
                w.writerow([r["question_no"], r["question"], r["run"], r["engine"], s])

    runs, seen, wording = {}, {}, {}
    for r in rows:
        runs.setdefault((r["question_no"], r["engine"]), set()).add(r["run"])
        for s in r["searches"]:
            key = (r["question_no"], norm(s))
            seen.setdefault(key, {}).setdefault(r["engine"], set()).add(r["run"])
            wording.setdefault(key, s)
    ranked = []
    for key, by in seen.items():
        qno = key[0]
        question = next(r["question"] for r in rows if r["question_no"] == qno)
        gpt, cl = len(by.get("chatgpt", ())), len(by.get("claude", ()))
        ranked.append([qno, question, wording[key], gpt, len(runs.get((qno, "chatgpt"), ())),
                       cl, len(runs.get((qno, "claude"), ())), gpt + cl])
    ranked.sort(key=lambda x: (x[0], -x[7], x[2]))
    with open("ranked.csv", "w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["question_no", "question", "search", "chatgpt_runs", "chatgpt_of",
                    "claude_runs", "claude_of", "runs_total"])
        w.writerows(ranked)

    for qno in sorted({x[0] for x in ranked}):
        top = [x for x in ranked if x[0] == qno][:3]
        print(f"\nq{qno} {top[0][1]}")
        for x in top:
            print(f"  ChatGPT {x[3]}/{x[4]}  Claude {x[5]}/{x[6]}  {x[2][:64]}")
    print(f"\n{len(rows)} calls, {sum(len(r['searches']) for r in rows)} searches logged, "
          f"{sum(1 for x in ranked if x[7] > 1)} came back in more than one run"
          + (f", {lost} unanswered" if lost else "") + ".")
    print(f"Cost of the calls in this folder: ${spent:.4f}")


if __name__ == "__main__":
    main()

The script sends each question to both engines with web search allowed but not forced, so whether a model searches at all is part of what it records. It keeps every response in a raw folder, never buys the same call twice, and stops before the next call could pass the budget you give it.

Step 3: Put your DataForSEO login in the terminal

In DataForSEO’s dashboard, open API Access and copy the API login and API password shown there. Open a terminal in your folder and paste these two lines, with your own values between the quotes:

export DATAFORSEO_LOGIN="your API login"
export DATAFORSEO_PASSWORD="your API password"

The two lines last until you close that terminal, so run the next two steps in the same window.

Step 4: Run the pilot

Run one question, once on each engine, before you spend anything on the rest:

python3 log_searches.py --pilot

Inside a minute you should see two lines, the searches, and a cost. This is ours:

Each line counts what one call returned: the searches the model ran, the citation marks in its answer (one per cited sentence, so a page cited three times counts three), and what the call cost. Below them sit the searches, each with the number of runs it appeared in on each engine.

If you get Refused: HTTP 401 instead, the login or password from step 3 is wrong, and nothing was bought.

Both APIs returned the searches in our pilot, and that is the question the whole procedure depends on. The rest of each response differs by engine, and the raw folder keeps all of it:

The pilot also showed the first misreading. ChatGPT searched our first question nearly as written. Claude searched for “ChatGPT recommendations startup visibility 2026” and for promotion in OpenAI’s GPT Store, which is where an app gets listed and not how a company gets recommended.

Step 5: Run all five rounds

Run the full set with a budget in dollars:

python3 log_searches.py --repeats 5 --budget 2.85

The script reuses the pilot’s two calls and buys the other 78. Our whole run, pilot to last call, took nineteen minutes. Our ceiling was $3, and we gave it 2.85 because it judges the next call by the most expensive one so far, so one unusually expensive call could take it slightly past the figure you set.

Twenty-three of our Claude calls came back with Refused: HTTP 500, and the script stopped each time. That is DataForSEO’s internal server error, and each reply said the call cost nothing, which our balance confirmed to the cent.

Run the same command again: it reads what it already bought from raw and carries on from the call that failed. We ran it twenty-four times to finish, and on the last one the window looked like this:

The last line is the whole bill: $1.3153 for 80 calls. Question 4 is missing from the summary because neither engine searched for it in any of its ten runs.

Keep the Searches That Come Back

The folder now holds every search the two models ran, and most of the wordings turned up once.

A search that comes back in four runs of five is the engine telling you what it needs to find. A search that comes back once is the model rewording itself.

The ranking matters more than any single answer, because the engine cites from the pages its searches return, and a search that keeps coming back decides which pages are in contention.

Step 6: Read ranked.csv

Open ranked.csv in Google Sheets or Excel and sort it by runs_total, largest first. Each row is one search for one question, with the number of runs, out of five, in which each engine ran it. Read the rows in three kinds:

Read across the questions next, because the pattern says how each engine read your buyer:

ChatGPT searched the buyer’s own words: 15 of its 20 searches were the question as typed. Claude never did. It rewrote every one of its 43 searches, sometimes adding the year, which is why the two engines matched on no search word for word.

Step 7: Group the wordings with Claude Code

Exact text undercounts. ChatGPT’s five searches for our first question are five slightly different sentences, so the sheet calls each a one-off, when they are one need that came back in every run. Grouping by meaning is the step a script does badly and a model does well. In the terminal, in your run folder, type claude to open Claude Code, then paste this prompt:

Read ranked.csv in this folder. Each row is a web search that ChatGPT or Claude ran before answering one of our buyer questions, with the number of runs (out of five) in which each engine ran it.

1. Group the searches that ask for the same thing in different words, within each question. Keep every original wording in its group.
2. For each group, count the runs on each engine. A run counts once per engine even if several wordings of the group appeared in it; use searches.csv to check.
3. Mark a group "recurring" if it came back in 3 or more of 5 runs on either engine, "occasional" at 2, and "one-off" at 1.
4. For each recurring or occasional group, write in one line the page a buyer would need to find for that search, and say what kind of page it is: list, comparison, how-to, pricing, or definition.
5. Write content-needs.csv with the columns question_no, group, wordings, chatgpt_runs, claude_runs, status, page_needed, page_type. Sort it by question_no, then recurring first.

Use only the searches in the files. Do not add searches, and do not merge groups across questions.

Ours turned 38 wordings into 14 needs, 8 of them recurring, and wrote content-needs.csv. The recurring ones:

We ran the prompt once, on Sonnet, and checked every row against searches.csv before using it. The grouping is the model’s judgment, and a wrong merge here would send a page in the wrong direction.

Step 8: Decide what each need gets

A recurring need gets one of four answers: a page of your own, a mention on the pages that came back for it, a fix to the words on your site, or nothing. The CSV cannot choose, because it cannot see your site, your prices, or which lists already rank. On a partnership a senior consultant makes this call with your team. This is how we read our own:

Vlad Shvets

Founder @ Empact Partners

The log changed what we do first on our own category. Claude searched for GEO consultancies in all five runs and read the same nine roundups each time, so the first month goes on getting into those nine, and a new page of ours comes after. Where ChatGPT read GEO as geography, the fix is the words on our own pages, and no amount of mention work reaches that.

Those answers split across two workstreams. The rewritten guide and the vocabulary are Content Marketing. The roundups are the mention half of GEO, where Existing Article Outreach, our method for this, gets a partner written into roundups the engines already read.

On a partnership, Claude Code drafts the brief for each page from the needs file. It publishes nothing, and a consultant decides what ships.

What the Search Log Cannot Tell You

The API is not the app. Our panel reads the ChatGPT app the way a buyer sees it, and across four weekly waves it recorded 148 cited sites for these eight questions, counting a site once per question.

Three of them were among the pages the ChatGPT API’s searches returned in our run, and ten among either engine’s. The app ran a newer model than the inexpensive one we called, so the gap is the surface and the model together, and the log cannot say how much of it is which.

DataForSEO draws the same line about its own products: its scraper is for what users see, and this API is for studying how models generate answers.

Rerun the same questions.txt in a month, in a new folder, and before reading anything else, check whether your own domain came back in the pages:

grep -l "empact.partners" raw/*.json

Ours prints nothing today. The first file name it prints is the first sign the work reached the engine’s search, before any answer names you.

On a partnership, this log runs beside the panel from the first month. A senior consultant picks the questions with your team, reads the ranking, and turns it into the roadmap’s first two lists: the pages to write and the pages to get named on. The standing report reruns both every month, because GEO moves over quarters.

If your buyers ask AI engines about your category and you want to know what those engines search before they answer, book a call with me, and we’ll see whether we could work together.