Which AI Crawlers Visit My Site: 25 Days of Real Access Logs

On this page, 9 sections

In 25 days of my site’s nginx access log, 12 user agents claimed to be AI or search crawlers and sent 19,214 requests. Of those, 3,830 (19.9 percent) went to paths no crawler has a reason to ask for, such as /proc/self/cmdline and /key.pem. That points to a scanner borrowing crawler names, not to the crawlers themselves.

The log runs from 17 September to 11 October 2026 and holds 653,445 lines. I read it on 11 October, read only, and kept aggregates only, no IP addresses. The content calendar said 30 days, but the file holds 25, so every figure below is 25 days of one site.

I could not check a single one of these hits against the vendors’ published IP ranges. The first field in my log is a Cloudflare edge address, not the visitor. More on that further down.

How I counted

I grouped requests by the name in the User-Agent header. That is the weakest possible identity, and the whole post is about why. A name in a log line is a claim, nothing more.

For each name I counted four things: total requests, requests to probe paths, requests for /robots.txt, and requests that returned a 200 on a non-probe path. A probe path is one that matched this pattern:

(__vite|/@fs/|/proc/|\.pem|\.env|firebase|cgi-bin|debug/vars|/api/config|info\.php|userfiles|serviceaccount|gc-service|\.git/|phpinfo|/actuator|/console|id_rsa|\.aws|docker|config\.json|\.sql|backup)

Real examples that matched: /__vite_rsc_findSourceMapURL, /@fs/var/run/secrets/kubernetes.io/serviceaccount/, /proc/self/cmdline, /userfiles, /key.pem, /firebase-config.json and /cgi-bin/php. This is a site that runs WordPress behind Cloudflare. None of those paths belongs to it.

The classification is a regex on the path, so it is crude. It will miss some scanning and could, in principle, flag an odd legitimate request. That is why I also ran two control agents, covered below.

Which AI crawlers showed up

These are the 12 names that claimed to be an AI product or a major search crawler. The 12 together account for 19,214 of the 653,445 lines, which is 2.9 percent of the log. The rest of the log is people and every other kind of bot.

User agent (claimed)	Requests	Probe-path requests	Probe share	robots.txt fetches
ClaudeBot	3,590	767	21.4%	64
OAI-SearchBot	3,453	283	8.2%	28
Googlebot	2,601	272	10.5%	8
PerplexityBot	1,811	504	27.8%	14
ChatGPT-User	1,490	263	17.7%	0
Applebot	1,354	247	18.2%	14
GPTBot	929	259	27.9%	0
YouBot	841	265	31.5%	2
DuckAssistBot	839	246	29.3%	12
cohere-ai	819	272	33.2%	0
MistralAI-User	761	232	30.5%	0
Google-Extended	726	220	30.3%	0

Window: 17 September to 11 October 2026, 25 days including partial days. Probe share is probe-path requests divided by total requests for that name, computed in Python from the counts.

OpenAI’s three names (GPTBot, OAI-SearchBot, ChatGPT-User) add up to 5,872 requests. ClaudeBot has 3,590, and the same file shows 48 for Claude-User and 1 for Claude-SearchBot. I left those two out of every total, because my notes record that the ClaudeBot figure already includes those tokens, and adding them would count some requests twice.

If you stopped here, the answer to “which AI crawlers visit my website” would be “all of them, a lot”. That answer is wrong, or at least unproven, and the probe column shows why.

The spoofed-crawler finding

Start with the control. Two undisguised, high-volume bots in the same log barely probe at all. bingbot sent 8,874 requests and only 33 matched the probe pattern, 0.4 percent. PetalBot sent 29,639 and only 14 matched, under 0.1 percent.

So probing is not a property of “any bot”. Something specific is happening under the AI names.

The same paths under every name

Across the 12 claimed agents, the probe requests drew from one shared set of 73 distinct paths. GPTBot’s 52 distinct probe paths all sit inside that set, and each of the other agents shares between 48 and 52 of them.

Real crawlers from different companies do not ask for the same Kubernetes service-account path and the same private key filename. A single scanner cycling through user-agent strings would.

The counts point the same way. Fourteen agents sit between 220 and 283 probe-path requests each, with a median of 257. That list includes meta-externalagent (274) and Amazonbot (246), whose probe share is low, 2.4 and 3.9 percent, only because their total volume is large. ClaudeBot (767) and PerplexityBot (504) run higher than that band, and I do not have an explanation for it.

This is inference, not proof. The simplest explanation I can find for the data is one scanner rotating user-agent strings. I cannot rule out other explanations without the visitor IPs.

“Google-Extended” cannot be Google’s Google-Extended

726 requests carried a user agent of the form Mozilla/5.0 (compatible; Google-Extended; +http://www.google.com/bot.html). Google’s crawler documentation lists Google-Extended as a robots.txt control token that has no separate HTTP user agent string of its own.

If that page is right, and I read it on 11 October 2026, then by definition these 726 requests did not come from a Google client using that token. I wrote the full explanation in Google-Extended: a token, not a crawler.

It is not a robots.txt decision

My live robots.txt names every one of these agents in a shared group with Allow: /. I read it on 11 October. So a probe path that came back 404 was not refused by a robots rule. The server simply had nothing there.

The RFC for the protocol says this plainly: robots.txt rules are not a form of access authorization. A scanner that wears a crawler’s name is not stopped by a rule written for the crawler. If you want the line-by-line on how groups match, my post on robots.txt for AI crawlers covers it.

What the robots.txt column adds

GPTBot sent 929 requests and fetched /robots.txt zero times. ClaudeBot fetched it 64 times, OAI-SearchBot 28, PerplexityBot 14, DuckAssistBot 12.

RFC 9309 says crawlers should not use a cached copy of robots.txt for more than 24 hours unless it is unreachable. A crawler working to that rule would have fetched the file many times in 25 days. A name that sends 929 requests and never asks for it fits the spoofing story.

It does not prove it. meta-externalagent (11,653 requests) and Amazonbot (6,293) also show zero robots.txt fetches, and I have not checked what Meta or Amazon document about fetching the file.

For ChatGPT-User (1,490 requests, zero fetches), OpenAI’s page says that because the actions are user-initiated, robots.txt rules may not apply, so a zero is expected there. Treat a zero as a prompt to look closer, not a verdict.

One more contrast, with a caveat. Of bingbot’s 8,841 non-probe requests, 8,329 returned a 200, which is 94.2 percent. For GPTBot it was 270 of 670, or 40.3 percent. My summary counts only the 200s and does not split the rest into redirects and errors, so I am not drawing a conclusion from the gap beyond noting that it exists.

What I could not check

The check that settles this is the IP address. Vendors publish ranges for their crawlers, and a real GPTBot request comes from one of them. Perplexity’s documentation, for example, links a published IP list for each of PerplexityBot and Perplexity-User.

I cannot run that check on this log. My site sits behind Cloudflare, and the first field in each line is a Cloudflare edge IP. Cloudflare’s documentation says the original visitor address arrives in an appended header called CF-Connecting-IP, and its page on restoring visitor IPs (last updated 23 April 2026) walks through logging it on Apache with mod_remoteip.

My server runs nginx and I did not change its log format in this session. So the plain statement is: the 25 days of data I have cannot say which hits were real. A later window, logged with the real IP, can.

Run the same count on your log

These two scripts assume the standard nginx combined log format, where splitting on double quotes puts the request in field 2 and the user agent in field 6. I syntax-checked both with bash -n and ran them against a four-line sample log. They are a starting point, not a full audit.

awk -F'"' '{ print $6 }' access.log \
  | grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|CCBot|bingbot' \
  | sort | uniq -c | sort -rn

awk -F'"' '
  $6 ~ /GPTBot/ {
    n++
    split($2, req, " ")
    if (req[2] ~ /(__vite|\/@fs\/|\/proc\/|\.pem|\.env|firebase|cgi-bin|phpinfo|\.git\/|id_rsa|\.aws|\.sql)/) p++
  }
  END { printf "%d requests, %d probe paths, %.1f%%\n", n, p, (n ? 100 * p / n : 0) }
' access.log

Swap GPTBot for any other name. If the probe share for a famous crawler is far above bingbot’s, you have the same pattern I do.

Questions people ask

Which AI crawlers visit my website?

You can list the names in your log, but a name is only a claim. On my site, 12 names claimed to be AI or search crawlers and about one in five of their requests went to probe paths. Without the visitor IP you cannot say how many were real.

How do I tell a real GPTBot from a fake one?

Match the requesting IP against the vendor’s published ranges. For that you need the real visitor IP in your log, which behind Cloudflare means restoring it from CF-Connecting-IP.

Does a high request count mean AI is using my content?

No. A request is a request. OpenAI’s own documentation separates a training crawler, a search crawler and a user-action fetcher, and the log line does not say which purpose a hit served.

Will robots.txt stop the fakes?

No. RFC 9309 states that these rules are not a form of access authorization. They ask honest crawlers to stay away. Anything that pretends to be one ignores them for free.

If you want the broader habit of reading the file for what scanners will not tell you, start with reading your WordPress access logs.

Do this today

Run the probe-share script against your own access log for GPTBot, then for bingbot. If GPTBot’s share is not close to bingbot’s, stop quoting crawler counts from that log until you are logging the real visitor IP.

Resources

• Google’s common crawlers documentation, read 11 October 2026: Google-Extended has no separate user agent string.
• RFC 9309, Robots Exclusion Protocol: rules are not access authorization, and the 24-hour cache guidance.
• Cloudflare: restoring original visitor IPs, read 11 October 2026, page last updated 23 April 2026.
• Perplexity crawlers documentation, read 11 October 2026: published IP lists for PerplexityBot and Perplexity-User.
• OpenAI crawlers and user agents documentation, read 11 October 2026.