Aditya Sharma

Verify the GPTBot IP Address: Real Crawler or Fake User Agent

On this page, 9 sections

To verify GPTBot, ignore the user agent and check the IP address. OpenAI publishes its GPTBot ranges as JSON at openai.com/gptbot.json. If the requesting IP sits inside one of those CIDR ranges, the request came from OpenAI’s published GPTBot network. If it does not, treat it as unverified, whatever the header says.

I read OpenAI’s crawler page and fetched the three range files on 11 October 2026. The GPTBot file held 18 ranges with a creationTime of 22 September 2026. Below are the exact scripts I used, and the output I got when I ran them.

Googlebot is verified a different way: a reverse DNS lookup, then a forward lookup that has to return the same address. Google documents both that method and a JSON range file. I ran both.

Why the user agent proves nothing

The User-Agent is a request header. Any client can send any string in it. Google’s own verification page exists because of this: it says verification is useful if you are concerned that spammers or other troublemakers are accessing your site while claiming to be from Google.

That is not hypothetical on my site. In 25 days of my access logs, a scanner appeared to be sending probe requests under a dozen crawler names, including a “Google-Extended” string that Google says does not exist as a user agent. I wrote up the numbers in AI crawlers in my access logs.

The same post explains why I could not take the next step on that data: my log records a Cloudflare edge address, not the visitor. Everything below needs the real client IP in your log.

Verify OpenAI’s crawlers by IP

OpenAI’s page lists a published IP file for each of its bots. These are the three that matter for a normal site, with what I measured on 11 October 2026.

BotFileRangesIPv4 addresses coveredcreationTime in file
GPTBotopenai.com/gptbot.json182,7042026-09-22
OAI-SearchBotopenai.com/searchbot.json392,5602026-01-02
ChatGPT-Useropenai.com/chatgpt-user.json23436,4962026-10-07

Range and address counts were computed in Python with the ipaddress module from the files as served on 11 October 2026. All three files use ipv4Prefix keys only.

The script below loads a range file from a URL or a local path and checks one or more IPs against it. It also handles ipv6Prefix entries, because Google’s file contains them. It exits with 1 if any address is not in the list, so you can use it in a pipeline.

#!/usr/bin/env python3
"""usage: verify_ip.py <ranges-json-url-or-path> <ip> [<ip> ...]"""
import ipaddress
import json
import sys
import urllib.request


def load(src):
    if src.startswith(("http://", "https://")):
        req = urllib.request.Request(
            src, headers={"User-Agent": "ip-check/1.0 (+https://adityaarsharma.com)"}
        )
        with urllib.request.urlopen(req, timeout=20) as r:
            data = json.load(r)
    else:
        with open(src) as f:
            data = json.load(f)
    nets = []
    for p in data["prefixes"]:
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return data.get("creationTime"), nets


def main():
    creation, nets = load(sys.argv[1])
    print(f"{len(nets)} ranges, creationTime {creation}")
    missing = 0
    for raw in sys.argv[2:]:
        ip = ipaddress.ip_address(raw)
        hit = next((n for n in nets if n.version == ip.version and ip in n), None)
        print(f"{raw}: {'IN ' + str(hit) if hit else 'NOT in list'}")
        missing += hit is None
    sys.exit(1 if missing else 0)


main()

I checked it with python3 -m py_compile and then ran it against the live file. 20.171.206.5 is inside the 20.171.206.0/24 range of the GPTBot list. 203.0.113.9 is in a range reserved for documentation, so it is a safe stand-in for a stranger.

$ python3 verify_ip.py https://openai.com/gptbot.json 20.171.206.5 203.0.113.9
18 ranges, creationTime 2026-09-22T02:00:07.000000
20.171.206.5: IN 20.171.206.0/24
203.0.113.9: NOT in list

The exit code was 1 because one address was missing. With only the first address it exits 0.

Check against the list for the bot in the header

The three files are not interchangeable. 104.208.184.200 is inside the ChatGPT-User list but not in the GPTBot list. A request that sends a GPTBot header from that address is not a verified GPTBot request, even though the IP belongs to OpenAI’s published space.

There is also overlap. The GPTBot and OAI-SearchBot files share 6 pairs of overlapping ranges (computed in Python on 11 October 2026). For an address in the overlap, the IP alone cannot tell you which of the two bots made the request, and the header is the only hint you have.

From log lines to a verdict

Once your access log records the real client address, three commands take you from a user agent to a list of unverified IPs. This assumes nginx’s combined format, where the IP is the first field and splitting on double quotes puts the user agent in field 6.

awk -F'"' '$6 ~ /GPTBot/ { split($1, f, " "); print f[1] }' access.log \
  | sort -u \
  | xargs python3 verify_ip.py https://openai.com/gptbot.json

I ran this on a four-line sample log with two GPTBot lines from 20.171.206.5 and 203.0.113.9. It printed one IN and one NOT in list, and exited 1. I checked the script with bash -n first.

If your site is behind Cloudflare, the field you need is not in the default log. Cloudflare’s documentation says the original visitor address arrives in a header called CF-Connecting-IP, and its page on restoring visitor IPs (last updated 23 April 2026) shows how to log it on Apache. For the background on reading logs at all, see reading your WordPress access logs.

Verify Googlebot with reverse and forward DNS

Google splits its clients into three types, and each has its own reverse DNS pattern and its own JSON file. I read the verification page on 11 October 2026. Its footer shows it was last updated on 20 March 2026.

TypeReverse DNS maskIP file
Common crawlers (Googlebot)crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.comcommon-crawlers.json
Special-case crawlers (AdsBot)rate-limited-proxy-***-***-***-***.google.comspecial-crawlers.json
User-triggered fetchersgoogle.com or gae.googleusercontent.com hostnamesuser-triggered-fetchers-google.json and user-triggered-fetchers.json

Google’s manual method has four steps. Run a reverse lookup on the IP from your log. Check that the name ends in googlebot.com, google.com or googleusercontent.com. Run a forward lookup on that name. Check that it returns the original IP.

The second lookup matters because a reverse record is set by whoever controls the address block. Anyone can point their own IP at a name that looks like Google’s. Only Google controls what googlebot.com names resolve to, so the forward lookup is what closes the loop.

#!/usr/bin/env bash
# usage: verify_googlebot.sh <ipv4>
ip="$1"
ptr=$(host "$ip" | awk '/domain name pointer/ {print $NF}' | head -n1 | sed 's/\.$//')
case "$ptr" in
  *.googlebot.com|*.google.com|*.googleusercontent.com) ;;
  *) echo "FAIL reverse lookup: ${ptr:-no PTR record}"; exit 1 ;;
esac
if host "$ptr" | grep -q "has address $ip\$"; then
  echo "OK $ip -> $ptr -> $ip"
else
  echo "FAIL forward lookup for $ptr"; exit 1
fi

I syntax-checked it with bash -n and ran it on 11 October 2026. It handles IPv4 only. The sample address 66.249.66.1 is the one in Google’s own documentation, and it passed. 8.8.8.8 is an address Google owns, but it is not a crawler, and it failed at the first step.

$ bash verify_googlebot.sh 66.249.66.1
OK 66.249.66.1 -> crawl-66-249-66-1.googlebot.com -> 66.249.66.1
$ bash verify_googlebot.sh 8.8.8.8
FAIL reverse lookup: dns.google

Google also documents matching the IP against its JSON range files instead. The same Python script works unchanged. On the same day, Google’s common-crawlers.json held 317 ranges with a creationTime of 9 October 2026, and 66.249.66.1 matched 66.249.66.0/27.

One detail worth knowing before you build a block rule. Google states that user-triggered fetchers ignore robots.txt rules, and that special-case crawlers may or may not respect them. Passing verification tells you who made the request. It does not tell you the request followed your robots.txt, which I broke down per crawler.

The other crawlers, and what I could not confirm

CCBot. Common Crawl documents reverse DNS on dedicated ranges (names under crawl.commoncrawl.org) and a range file at index.commoncrawl.org/ccbot.json. The file notes that forward-confirmed reverse DNS is also recommended for IPv4.

I ran its sample, host 18.97.14.84, and got a pointer to 18-97-14-84.crawl.commoncrawl.org, then the matching address on the forward lookup. The page also warns that crawlers falsely identify themselves as CCBot.

Perplexity. Its documentation links published IP files for PerplexityBot and Perplexity-User, and suggests WAF rules that combine a user-agent match with an IP-range match. I read the docs but did not fetch or test those files.

Anthropic. The support page I fetched on 11 October 2026 names ClaudeBot, Claude-User and Claude-SearchBot. The page says a request from an IP address on a list it links to is coming from Anthropic, but the link did not survive my text extract, so I have not tested that list and cannot give you a verified method for those three.

That is “not shown on the page I read”, not “does not exist”. The page does say that blocking Anthropic’s IP addresses may not persistently guarantee an opt-out, because it stops the bot reading your robots.txt.

Limits of an IP check

  • Lists change. The ChatGPT-User file was created 15 days after the GPTBot file (7 October against 22 September 2026). Fetch the file on a schedule, and do not paste ranges into a firewall once and forget them.
  • A match proves origin, not purpose. It says the request came from the published network. It does not say what the content will be used for.
  • It needs the real IP. Behind a CDN or proxy, a default log records the proxy. A check against vendor ranges on that field returns only mismatches.
  • Some bots are documented only as user agents. For those you can note the claim, but not verify it.

Questions people ask

How do I verify a GPTBot IP address?

Fetch OpenAI’s published GPTBot file and check whether the IP falls inside any listed CIDR range. The script above does it in one command.

Is a reverse DNS lookup enough?

Not on its own. Google’s method requires the forward lookup to return the same IP. Common Crawl recommends the same forward-confirmed check.

Will robots.txt stop a fake GPTBot?

No. RFC 9309 says the rules are not a form of access authorization. A client pretending to be GPTBot can ignore them.

Can I block unverified GPTBot traffic automatically?

The logic is simple: user agent contains GPTBot and IP is not in the list. Perplexity’s documentation shows the mirror-image pattern for an allow rule. I have not tested a block rule on a live WAF, so check it on a staging site first.

Do this today

Open your access log and look at the first field of one line. If it is your CDN’s address, fix the log format before anything else. If it is the visitor, run the three-command pipeline for GPTBot and read the NOT in list lines.

Resources

Tell me where I am wrong

Your email is not published and I do not add it to any list. Corrections with a source are the ones I act on fastest.

More in AI

Every piece in AI

  1. ClaudeBot, Claude-User, Claude-SearchBot: What Each Crawler Does Anthropic documents three crawlers: ClaudeBot for training, Claude-User for user requests, Claude-SearchBot for search. What each does and how to control it. AI 6 min
  2. Which AI Crawlers Visit My Site: 25 Days of Real Access Logs I counted 12 AI and search crawler names in 25 days of nginx logs. About one in five of their requests hit probe paths,… AI 8 min
  3. Claude Code Hooks Explained: A Real 35-Hook Setup, Tested Claude Code hooks run at fixed lifecycle events and can block actions. How events, matchers and exit codes work, plus my real 35 hooks… AI 8 min
  4. GPTBot vs OAI-SearchBot vs ChatGPT-User: What Each One Does GPTBot trains, OAI-SearchBot feeds ChatGPT search, ChatGPT-User fetches on request. The user agents, robots.txt rules and IP lists, read 11 October 2026. AI 6 min
  5. How to Block AI Crawlers in WordPress (robots.txt Is Not a Lock) One robots.txt group per AI crawler token, a tested WordPress snippet, what each vendor documents, and why a physical robots.txt file can make your… AI 9 min
  6. What Is an MCP Server? I Ran One and Logged Every Message An MCP server gives an AI app tools, data and prompts over JSON-RPC. I ran a real one over stdio on 11 October 2026… AI 8 min