Aditya Sharma

GPTBot vs OAI-SearchBot vs ChatGPT-User: What Each One Does

On this page, 8 sections

GPTBot is OpenAI’s training crawler. OAI-SearchBot is the one that decides whether you appear in ChatGPT search results. ChatGPT-User is a fetcher that runs when a person asks ChatGPT about a page. They are separate user agents, they have separate robots.txt tokens, and blocking one does not block the others.

The line most sites get wrong is this one: blocking GPTBot says your content should not train OpenAI’s foundation models, and it does nothing to your visibility in ChatGPT search. That job belongs to OAI-SearchBot.

I read OpenAI’s bot documentation on 11 October 2026 and pulled the published IP lists for three of the bots the same day. Numbers below come from those two sources.

The four user agents

OpenAI’s page now lists four. Each has its own purpose, and each has its own example user agent string. For GPTBot and OAI-SearchBot, OpenAI notes the version number in the string can change, so match on the name, not on 1.4.

  • GPTBot (GPTBot/1.4 in the example string): crawls content that may be used in training generative AI foundation models. Disallowing it indicates a site’s content should not be used for training.
  • OAI-SearchBot (OAI-SearchBot/1.4): surfaces websites in ChatGPT’s search features. Sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links.
  • ChatGPT-User (ChatGPT-User/1.0): used for certain user actions in ChatGPT and Custom GPTs. When a user asks a question, it may visit a page. OpenAI says it is not used for crawling the web automatically.
  • OAI-AdsBot (OAI-AdsBot/1.0): validates pages submitted as ads on ChatGPT. It only visits submitted pages, and OpenAI says the data it collects is not used to train foundation models.

The opening paragraph of the page sets the frame. Each setting is independent of the others. Their example: allow OAI-SearchBot so you appear in search, and disallow GPTBot so your content is not used for training.

Which robots.txt rules apply to which bot

This is where the three-way split becomes practical.

GPTBot and OAI-SearchBot respond to robots.txt. OpenAI also says that if you allow both, it may use the results from one crawl for both uses, to avoid duplicate crawling. For search, it can take about 24 hours from a robots.txt update for the systems to adjust.

ChatGPT-User is different. OpenAI’s wording is that because these actions are initiated by a user, robots.txt rules may not apply. It adds that ChatGPT-User is not used to determine whether content may appear in Search, and tells you to use OAI-SearchBot in robots.txt for search opt-outs and automatic crawl.

So a Disallow aimed at ChatGPT-User is a request, not a guarantee. If you need to stop it, you need a control that works at the server or CDN, not in a text file.

Three policies and the file for each

Keep out of training, stay in ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Stay out of both training and search:

User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /

Allow everything, with the policy visible in the file:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /

One catch applies to all three: once a crawler finds a group naming it, it obeys only that group and ignores *. If you have Disallow: /wp-json/ under * and then list GPTBot in a group with only Allow: /, GPTBot is now allowed to fetch that path.

I found exactly this on my own site, and wrote it up in robots.txt for AI crawlers: the user agents that matter.

Verifying that a request is really OpenAI

A user agent string is a text field, and anyone can send it. OpenAI publishes IP ranges for each bot as a JSON file, which is the check that holds up.

I fetched three of them on 11 October 2026:

  • gptbot.json: 18 prefixes, file dated 22 September 2026.
  • searchbot.json: 39 prefixes, file dated 2 January 2026.
  • chatgpt-user.json: 234 prefixes, file dated 7 October 2026.

The creation time is read from the creationTime field inside each file. The count is the length of the prefixes array.

The spread is itself informative. ChatGPT-User has more than 12 times as many published ranges as GPTBot (234 against 18, my arithmetic). That fits a fetcher that runs from many places on behalf of users, rather than one training crawler. I am inferring that from the counts; OpenAI does not say it.

Here is a short script that checks an IP from your log against a downloaded list. It handles both IPv4 and IPv6 key names:

import ipaddress, json, sys

data = json.load(open(sys.argv[1]))   # e.g. gptbot.json
ip = ipaddress.ip_address(sys.argv[2])

for p in data["prefixes"]:
    cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
    if cidr and ip in ipaddress.ip_network(cidr):
        print("match", cidr)
        break
else:
    print("no match")

The prefix list changes, so do not hard-code it into a firewall rule once and forget it. OpenAI’s own recommendation for search is to allow both the bot and its published IP ranges.

Finding them in your logs

If you can read raw access logs, match the name. This counts requests per OpenAI bot in a standard combined log:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot" access.log \
  | grep -oE "(GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot)" \
  | sort | uniq -c | sort -rn

Remember that OpenAI may add a robots.txt marker to the user agent when it fetches your robots.txt file, so that request will look slightly different from the page fetches. It is a convenience for people whose logs do not show the path. For more on reading logs on a WordPress host, see reading WordPress access logs.

I could not run that command against adityaarsharma.com for this post. The site sits behind Cloudflare and I did not have its raw logs in this session, so I am not going to tell you what share of my traffic these bots are.

Questions people ask

What is the GPTBot user agent?

The example string in OpenAI’s documentation is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. The robots.txt token is GPTBot.

Does blocking GPTBot remove me from ChatGPT?

Not from search. OpenAI says disallowing GPTBot indicates content should not be used to train foundation models, and that OAI-SearchBot is the one that governs ChatGPT search results.

Should I allow ChatGPT-User?

That depends on whether you want a person’s question to be able to pull in your page. OpenAI says robots.txt rules may not apply to it, so allowing or disallowing it in the file is not a reliable control either way.

How long does a robots.txt change take?

For search, OpenAI says it can take about 24 hours from a site’s robots.txt update for its systems to adjust.

Reading the split as a decision

Three questions cover most sites.

Do you want your pages used to train OpenAI’s models? That is a GPTBot decision and nothing else. Do you want your pages to appear in ChatGPT search answers?

That is an OAI-SearchBot decision. Do you want a person who pastes your URL into ChatGPT to get an answer from the page? That is ChatGPT-User, and robots.txt is not a reliable lever for it.

Because the settings are independent, there is no “block OpenAI” line. There are three separate choices, and the file has to say each one. OAI-AdsBot only matters if you submit pages as ads on ChatGPT.

Do this today

Open your robots.txt and decide your policy for GPTBot and OAI-SearchBot separately, in two lines of plain English. Then check that the file does what you wrote. If it names both in one group, the policy is the same for both, which may not be what you meant.

Resources

Tell me where I am wrong

Your email is not published and I do not add it to any list. Corrections with a source are the ones I act on fastest.

More in AI

Every piece in AI

  1. How to Block AI Crawlers in WordPress (robots.txt Is Not a Lock) One robots.txt group per AI crawler token, a tested WordPress snippet, what each vendor documents, and why a physical robots.txt file can make your… AI 9 min
  2. What Is an MCP Server? I Ran One and Logged Every Message An MCP server gives an AI app tools, data and prompts over JSON-RPC. I ran a real one over stdio on 11 October 2026… AI 8 min
  3. Google-Extended User Agent: It Is a Token, Not a Crawler Google-Extended has no user agent string, so it never shows in your logs. What the token controls for Gemini, what it leaves alone in… AI 7 min
  4. AI Memory With Tagging: Tags, Embeddings and Graph Edges Are Not Interchangeable MemPalace ships five backends and the default one has a dash in the Namespaces column. Which tools expose tags, and where a tag beats… AI 10 min
  5. AI Memory for Writers: What Survives the Round Trip This category was built for code. One project targets prose, it has 66 stars, and the test that eliminates the rest takes two minutes. AI 9 min
  6. codebase-memory-mcp vs Graphify: What Each One Actually Stores Both are alive, both refuse a vector database, and both were pushed to this week. The split is who owns the graph file. AI 10 min