How To Check Which AI Bots Really Read Your Site

The steps below are for the marketing leader who has been told that AI engines cannot read the company’s site, or that they read it constantly, and who wants proof from the logs before anyone edits robots.txt. The usual proof is a count of GPTBot in the logs, and the name in a user agent is free to type.

Empact Partners runs this check in the first week of a technical audit. As a consultancy, we run go-to-market for B2B software teams, and the check belongs to our SEO work: Search Everywhere Optimization, which covers Google, the AI engines and the communities they quote. You end with a week of evidence and a fix list naming who changes what.

Each request that names an AI agent is checked against its operator’s published address list, then read beside what it fetched. On our own domains, for the week of 23 to 29 October, 6,902 of the 13,442 requests that named an AI agent were provably not that crawler, including all 4,497 that asked for a password or config file.

What You Need Before You Start

That week of evidence tells you which AI crawlers reached your site, whether they were real, and what they got. It does not tell you whether any AI engine cites you, which is a different measurement.

If your site runs on Webflow or Vercel with no CDN of your own in front, the host is no help: Webflow, which hosts our own site, gives us no request log at all, and Vercel gave us none we could export for a past week, because its log drain records from the day you add it.

Route One, Cloudflare: Pull the Week and Check Every Name

Step 1: Create a read-only token

In Cloudflare, open My Profile, then API Tokens, and create a custom token with two zone permissions: Analytics Read and Zone Read, for the zone you are checking. Nothing on this page writes to Cloudflare, so the token never needs more.

If the script in Step 3 answers “does not have permission”, the token is missing Analytics Read on that zone. An account-level token of ours failed exactly that way. A user token carrying the two zone permissions worked first time.

Step 2: Find your zone ID

Paste your token into the first line and your domain into the second, then run both in a terminal. The second prints a 32-character zone ID.

export CF_API_TOKEN=paste-your-token-here
curl -s -H "Authorization: Bearer $CF_API_TOKEN" "https://api.cloudflare.com/client/v4/zones?name=example.com" | python3 -c "import sys, json; print(json.load(sys.stdin)['result'][0]['id'])"

Step 3: Run the week through the script

The script carries the current list of AI agents, read at each operator’s own page on 30 October, with each operator’s address list attached where one is published. Twenty-six agents from eleven companies can show up in a log, and twenty of them come with a published list.

Two names you will see in robots.txt advice never appear in a log. Google says Google-Extended “doesn’t have a separate HTTP request user agent string”, and Apple says Applebot-Extended “does not crawl webpages”, so a zero for either means nothing. Google’s AI answers read what Googlebot fetched.

Save the script below as ai_bot_week.py in an empty folder. It asks Cloudflare for every request in your window that named an agent or that Cloudflare itself labelled as AI, downloads each operator’s address list, and writes ai-bot-week.csv: one row per agent, path, status and verdict, with no addresses in it.

"""One week of AI-agent requests, each checked against the operator's own published address list.

Cloudflare:  CF_API_TOKEN=your-token python3 ai_bot_week.py cloudflare ZONE_ID 2026-10-23 2026-10-30
Railway:     python3 ai_bot_week.py railway host-week.jsonl
Writes ai-bot-week.csv (no IP addresses in it) and prints the week by agent.
Agent list read at each operator's own page on 30 October 2026. Re-read them before you trust it.
"""
import csv, datetime, ipaddress, json, os, re, sys, urllib.request
from collections import Counter, defaultdict

G = "https://developers.google.com/static/crawling/ipranges/"
AGENTS = {  # token: (operator, what it does, the operator's address list or None)
    "GPTBot": ("OpenAI", "training", "https://openai.com/gptbot.json"),
    "OAI-SearchBot": ("OpenAI", "search index", "https://openai.com/searchbot.json"),
    "ChatGPT-User": ("OpenAI", "user fetch", "https://openai.com/chatgpt-user.json"),
    "OAI-AdsBot": ("OpenAI", "ads check", "https://openai.com/adsbot.json"),
    "ClaudeBot": ("Anthropic", "training", "https://claude.com/crawling/bots.json"),
    "Claude-User": ("Anthropic", "user fetch", "https://claude.com/crawling/bots.json"),
    "Claude-SearchBot": ("Anthropic", "search index", "https://claude.com/crawling/bots.json"),
    "PerplexityBot": ("Perplexity", "search index", "https://www.perplexity.ai/perplexitybot.json"),
    "Perplexity-User": ("Perplexity", "user fetch", "https://www.perplexity.ai/perplexity-user.json"),
    "Google-CloudVertexBot": ("Google", "owner-requested crawl", G + "special-crawlers.json"),
    "Google-GeminiNotebook": ("Google", "user fetch", G + "user-triggered-fetchers.json"),
    "Google-Agent": ("Google", "user agent", G + "user-triggered-agents.json"),
    "Applebot": ("Apple", "search index", "https://search.developer.apple.com/applebot.json"),
    "meta-externalagent": ("Meta", "training", None),
    "meta-webindexer": ("Meta", "search index", None),
    "meta-externalfetcher": ("Meta", "user fetch", None),
    "meta-externalads": ("Meta", "ads", None),
    "Amazonbot": ("Amazon", "training", "https://developer.amazon.com/amazonbot/ip-addresses/"),
    "Amzn-SearchBot": ("Amazon", "search index", "https://developer.amazon.com/amazonbot/searchbot-ip-addresses/"),
    "Amzn-User": ("Amazon", "user fetch", "https://developer.amazon.com/amazonbot/live-ip-addresses/"),
    "CCBot": ("Common Crawl", "training", "https://index.commoncrawl.org/ccbot.json"),
    "DuckAssistBot": ("DuckDuckGo", "user fetch", "https://duckduckgo.com/duckassistbot.json"),
    "MistralAI-User": ("Mistral", "user fetch", "https://mistral.ai/mistralai-user-ips.json"),
    "MistralAI-Index": ("Mistral", "search index", "https://mistral.ai/mistralai-index-ips.json"),
    "MistralAI-Training": ("Mistral", "training", None),
    "Bytespider": ("ByteDance", "training", None),
}
INSIDE_ONLY = {"Anthropic"}  # its list proves a request is Anthropic's, never that one is not


def address_lists():
    nets = defaultdict(list)
    for url in {a[2] for a in AGENTS.values() if a[2]}:
        req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
        text = urllib.request.urlopen(req, timeout=60).read().decode("utf-8", "ignore")
        text = re.sub(r"(?is)<script.*?</script>", "", text)  # Amazon's pages print YOUR address in a script
        for m in re.findall(r"[0-9a-fA-F:.]{7,}/\d{1,3}|\b\d{1,3}(?:\.\d{1,3}){3}\b", text):
            try:
                net = ipaddress.ip_network(m, strict=False)
            except ValueError:
                continue
            for op in {a[0] for a in AGENTS.values() if a[2] == url}:
                nets[op].append(net)
    return nets


def verdict(op, ip, cloudflare_label, nets, operators_at_address):
    if cloudflare_label or any(ipaddress.ip_address(ip) in n for n in nets.get(op, [])):
        return "verified"
    if nets.get(op) and op not in INSIDE_ONLY:
        return "disproven"
    if operators_at_address > 1:  # one address cannot be two companies' crawler
        return "disproven"
    return "unresolved"


def cloudflare(zone, start, end):
    q = """query($z:String!,$a:Time!,$b:Time!,$f:[ZoneHttpRequestsAdaptiveGroupsFilter_InputObject!]){
      viewer{zones(filter:{zoneTag:$z}){httpRequestsAdaptiveGroups(limit:10000,
        filter:{datetime_geq:$a,datetime_lt:$b,OR:$f}){count dimensions{date clientRequestHTTPHost
        clientRequestPath userAgent clientIP edgeResponseStatus verifiedBotCategory}}}}}"""
    f = [{"userAgent_like": f"%{t}%"} for t in AGENTS] + [{"verifiedBotCategory_like": "AI%"}]
    day, rows = datetime.date.fromisoformat(start), []
    while day < datetime.date.fromisoformat(end):
        v = {"z": zone, "a": f"{day}T00:00:00Z", "b": f"{day + datetime.timedelta(days=1)}T00:00:00Z", "f": f}
        req = urllib.request.Request("https://api.cloudflare.com/client/v4/graphql", method="POST",
                                     data=json.dumps({"query": q, "variables": v}).encode(), headers={
                                         "Authorization": "Bearer " + os.environ["CF_API_TOKEN"],
                                         "Content-Type": "application/json"})
        res = json.load(urllib.request.urlopen(req, timeout=120))
        if res.get("errors"):
            sys.exit(res["errors"])
        groups = res["data"]["viewer"]["zones"][0]["httpRequestsAdaptiveGroups"]
        if len(groups) == 10000:
            print(f"{day}: 10,000 groups, the limit; split the day into hours")
        for g in groups:
            d = g["dimensions"]
            rows.append((d["date"], d["clientRequestHTTPHost"], d["clientRequestPath"], d["edgeResponseStatus"],
                         d["userAgent"], d["clientIP"], d["verifiedBotCategory"], g["count"]))
        day += datetime.timedelta(days=1)
    return rows


def railway(path):
    rows = []
    for line in open(path):
        if line.startswith("{"):
            r = json.loads(line)
            rows.append((r["timestamp"][:10], r["host"], r["path"].split("?")[0], r["httpStatus"],
                         r.get("clientUa", ""), r["srcIp"], "", 1))
    return rows


def main():
    rows = cloudflare(*sys.argv[2:5]) if sys.argv[1] == "cloudflare" else railway(sys.argv[2])
    nets, out, week, claims = address_lists(), Counter(), defaultdict(Counter), defaultdict(set)
    for r in rows:
        token = max((t for t in AGENTS if t.lower() in r[4].lower()), key=len, default=None)
        if token:
            claims[r[5]].add(AGENTS[token][0])
    for day, host, path, status, ua, ip, label, n in rows:
        token = max((t for t in AGENTS if t.lower() in ua.lower()), key=len, default=None)
        if token:
            op, kind, _ = AGENTS[token]
        elif label:
            op, kind = "(Cloudflare: " + label + ")", "unnamed"
        else:  # no AI name: keep it only if it came from an operator's own addresses
            op = next((o for o, ns in nets.items() if any(ipaddress.ip_address(ip) in x for x in ns)), None)
            if not op:
                continue
            kind = "unnamed, from " + op + "'s addresses"
        v = verdict(op, ip, label, nets, len(claims[ip]))
        out[(day, host, path, status, token or ua[:60], op, kind, label, len(claims[ip]), v)] += n
        week[token or op][v] += n
        week[token or op][f"status {status}"] += n
    with open("ai-bot-week.csv", "w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["day", "host", "path", "status", "agent", "operator", "kind", "cloudflare_label",
                    "operators_at_address", "verdict", "requests"])
        w.writerows(k + (n,) for k, n in sorted(out.items()))
    for agent, c in sorted(week.items(), key=lambda x: -sum(v for k, v in x[1].items() if "status" not in k)):
        total = sum(v for k, v in c.items() if not k.startswith("status"))
        print(f"{agent:22} {total:7}  " + "  ".join(f"{k} {v}" for k, v in sorted(c.items())))


main()

Run it from the same folder with your zone ID, the first day of your window, and the day after its last:

python3 ai_bot_week.py cloudflare ZONE_ID 2026-10-23 2026-10-30

The script finished in under a minute per domain for us and prints the week by agent. If a day prints “10,000 groups, the limit; split the day into hours”, that day’s rows stopped at the most one query returns and its counts are short. Splitting means editing the query’s time window, because the script works in whole days. No day of ours came near it.

Step 4: Read the names against the verdicts

Every row carries one of three verdicts, and they are the first thing to read in the sheet.

Our week came out like this:

Applebot carries our verified total, 5,598 of the 6,438. Take it out and 840 of 7,368 named requests verify. Name by name, the share that verified looks like this.

Read the counts under the names. Eleven agents from seven companies each landed between 480 and 590 requests in the same week. Real crawlers from unrelated companies do not arrive in matching batches. A scanner rotating through a list of names does.

The operators_at_address column says so in numbers: 6,867 of our named requests came from addresses that claimed two or more companies’ crawlers.

Read that column before any total. Once the scanners are out, what is left is a much shorter list of real crawlers, and the questions about those are the ones worth a meeting.

Step 5: Read what each group asked for and what it got

Group the rows by the kind of path they asked for. The verdict and the path together tell you more than either alone.

Real crawlers read robots.txt and pages. The requests that wore their names went hunting for .env files and admin panels, and not one request for a secret file verified. Where an operator publishes no list, the path is the second test: a request calling itself Bytespider that asks for /@fs/home/ubuntu/.aws/credentials, as one did in our week, is not a crawler, whatever the list can say.

Then read the status column. Our domains are retired publishing sites that redirect everyone, so every verified crawler got the same redirect a person gets. The 229 blocked requests were all disproven, nearly all on app and admin paths of domains with no app. Every bot setting on our zones is off, so the block was the platform’s own protection.

On the Free plan the analytics API would not tell us which rule fired, and the path and the verdict were enough to leave it alone. The counts are Cloudflare’s estimates from a sampled dataset: running the same week twice moved single rows by one request, so quote them rounded.

Step 6: Read the agents your list does not know

The script also pulls every request Cloudflare itself labels as AI, named or not, because a list kept by hand is always somebody’s update behind.

In our week 383 such requests carried names our list leaves out: PetalBot, Reflectionbot, ShapBot, and GoogleOther, which Google describes as a generic crawler its product teams may use. Each is a decision for the fix list, allow or ignore, rather than a fault.

One request carried Cloudflare’s “AI Assistant” label and the user agent of an ordinary Chrome browser on Linux, with no agent name at all. A filter on names never sees it. It is the only AI agent presenting as a browser that this method can show you, and it shows only the ones Cloudflare recognized.

Route Two, a Host’s Raw Logs: Prove the Log Can See an AI Read

Cloudflare sees only what it sits in front of. If your host keeps raw request logs with the user agent and the client address, the same script reads them, and the same three verdicts come out.

Step 7: Export the host’s week

On Railway, from a terminal signed in with its command-line tool, pull the week of HTTP logs for the service. Change the project ID and the service name to yours.

railway logs --http --project YOUR_PROJECT_ID --service web --environment production --since 2026-10-23T00:00:00Z --until 2026-10-30T00:00:00Z --lines 5000 --json > host-week.jsonl

Railway refused --lines 10000 with “Error in beforeLimit - Invalid input”, so a busy week is pulled a few days at a time. Its logs also belong to a deployment: if the service was redeployed midweek, the days before it sit with the older deployment.

Step 8: Run the same script on it

python3 ai_bot_week.py railway host-week.jsonl

Our host route ran on one of our own small internal tools: 451 requests in the week, and not one named an AI agent or came from an operator’s address.

The reason was in the same log. The tool serves a robots.txt that disallows everything, on purpose, and robots.txt was fetched twice. A polite crawler reads that and leaves, which is what we want for an internal tool and the opposite of what a marketing site wants.

Step 9: Send one AI read and find it in the log

A zero from a host log proves nothing until you have watched that log record an AI read. Make up a path nobody links to, ask an AI assistant to open that exact address, and search the log for the path.

We asked Gemini, through Google’s Antigravity command-line tool, to open such a path on our tool. Within a second of the job starting the log held three requests, and none of them named a crawler from any operator’s page:

None of the three was preceded by a robots.txt request. The address check found the first one, and the script keeps any request from an operator’s own addresses even when its name is unknown, so Google’s line sits in the sheet as verified.

The other two came from our own machine under the name of a generic code library, and nothing in a log marks them as AI. A log count of AI reads is a floor.

Building that check, we found a trap worth knowing before you write your own. Amazon publishes its address lists as web pages, and each page also prints the visitor’s own address inside a script, so a careless scrape adds you to Amazon’s crawlers. Ours did, for one run, until the script learned to drop scripts before reading addresses.

What Reddit says

“anthropic documents that claude-user respects robots.txt, and thats what i saw. it read robots.txt before the page, and when i disallowed a path it refused without sending a single request.” — u/hunxai69, r/TechSEO, Oct 2026

“Checked again now with regular in-app chat. History doesnt show /robots.txt fetch, straight up the given urls.” — u/alvares169, r/TechSEO, Oct 2026

Two people tested the same assistant in the same thread and got opposite behavior, which is why the fetch in Step 9 is yours to run rather than ours to report.

Turn the Sheet Into a Fix List

Our week’s fix list, written straight from the sheet:

A user agent is a name anyone can type. The address on the operator’s own list is the part of a request a scanner cannot borrow.

Half of that list says leave it alone, and that is the usual shape. Separating working-as-intended from a fault is most of the skill, and a fix list that treats every flagged row as a fault sends engineering after scanners for a week.

So we changed nothing on our sites. Nothing verified was refused, every block hit a disproven request, and the empty host is its own robots.txt doing its job. The two real changes were to the report and to the list, and neither needs a deploy.

On a live marketing site the list usually holds rows ours did not:

Vlad Shvets

Founder @ Empact Partners

Check the address before a number goes in a report and before a block goes in a rule. On our own domains half the requests that named an AI crawler were not that crawler, and every request for a password file was one of them. A team that tunes its site to the raw count is tuning it for scanners.

In the four partner site audits our consultants have run since August that read robots.txt, none had shut out an AI crawler by name.

What we found was duller: a robots.txt rule closing a whole section to every crawler, a site with no robots.txt at all, a pricing page whose price appeared only after scripts ran, and canonicals pointing at addresses that redirect. The toggle everybody fears is rarely the culprit. The inherited rule nobody reread usually is.

Our partnerships on this start the same way. A senior consultant runs this week of logs and requests your pages as each AI crawler, writes the fix list with owners, and hands the CDN rows to your engineers with the check that proves each one landed. The rule changes, the robots.txt edits and the template fixes stay with the people who own them.

What a Clean Sheet Cannot Tell You

A clean sheet says the engines’ crawlers reached your pages and what they got back. It says nothing about whether any engine names you when a buyer asks about your category, because a crawl is not a citation. A page can be read every day and quoted never.

That second question is our Generative Engine Optimization work, and it is measured differently: our Empact Panel asks ChatGPT, Google AI Mode and Claude the same buyer questions every week and records who gets named. Run the log check first, because there is no point measuring citations of a page the engines cannot reach.

If you want this check run on your site and the fix list owned through to the last row, pick a time in my calendar, and we can work out together whether we are the right people for it.