ClaudeBot, Claude-User, Claude-SearchBot: What Each Crawler Does
On this page, 9 sections
ClaudeBot is Anthropic’s training crawler. Anthropic lists two more: Claude-User, which fetches pages when someone asks Claude a question, and Claude-SearchBot, which analyses content to improve search results. Each is controlled with its own robots.txt token, and Anthropic says it honours robots.txt “do not crawl” signals.
If you only block ClaudeBot, you are saying your future content should be excluded from model training. You are not blocking Claude’s user-directed fetching or its search indexing. Those are the other two.
I read Anthropic’s help article on its crawlers on 11 October 2026. This post is built from that page, and I mark where it is silent.
The three Anthropic bots
The article describes three robots and what happens if you disable each one.
- ClaudeBot helps enhance the utility and safety of Anthropic’s generative AI models by collecting web content that could potentially contribute to training. When a site restricts it, that signals the site’s future materials should be excluded from training datasets.
- Claude-User supports Claude users. When someone asks Claude a question, it may access websites using this agent. Disabling it prevents Anthropic’s system from retrieving your content in response to a user query, which the article says may reduce your visibility for user-directed web search.
- Claude-SearchBot navigates the web to improve search result quality. Disabling it prevents indexing your content for search optimisation, which the article says may reduce your visibility and accuracy in user search results.
That maps cleanly onto the OpenAI split: training, user-initiated fetching, search. If you have read GPTBot vs OAI-SearchBot vs ChatGPT-User, the structure is the same. The difference is in the details of control, below.
What the page does not tell you
I read the article for user agent strings and found none. It names the three robots and the robots.txt behaviour, but it does not print full HTTP user agent strings the way OpenAI and Perplexity do. I am not going to invent them.
What that means in practice: match on the bot name inside the user agent field of your logs (ClaudeBot, Claude-User, Claude-SearchBot). Then treat the name as a claim, not proof, because any client can send it.
The article also says that a crawler with a source IP address on Anthropic’s published list is coming from Anthropic. So the verification path is: match the name, then compare the source IP with that list. I did not download the list for this post, so I cannot give you a range count the way I can for OpenAI.
Old tokens you may have in robots.txt
Some generated robots.txt files name anthropic-ai and Claude-Web. Neither appears on the three-bot list in the article I read. That is all I can say from that page. I cannot tell you what those older tokens did, only that the current documentation does not list them. If you are keeping them, add the three current names alongside them.
How to control them in robots.txt
Anthropic’s instruction for a full block is the standard one:
User-agent: ClaudeBot
Disallow: /It says to put this in the robots.txt in your top-level directory, and to do it for every subdomain you want to opt out. A subdomain has its own robots.txt, so a rule on example.com does nothing for blog.example.com.
To keep out of training and stay available to Claude’s search and user requests:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /To block all three:
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /Crawl-delay is supported
This is the detail that differs from most vendors. Anthropic says its bots support the non-standard Crawl-delay extension, and gives this example:
User-agent: ClaudeBot
Crawl-delay: 1It is non-standard, so do not assume other crawlers read it. For Anthropic’s bots, it is a documented way to slow the crawl without blocking it. The article says it aims to respect Crawl-delay where appropriate, so treat it as a request with a stated intent, not a rate limiter.
Do not block by IP to opt out
Anthropic’s article warns against this. It says alternate methods, such as blocking the IP addresses its bots operate from, may not work correctly or persistently guarantee an opt-out, because doing so impedes its ability to read your robots.txt file.
The sensible reading: use robots.txt for the opt-out. Use IP filtering for verification and abuse handling, not as your policy.
The group-matching trap, again
A crawler that finds a group naming it obeys only that group. The * group stops applying. If you put ClaudeBot in a long shared group with Allow: /, every Disallow under * stops applying to it.
I found this on my own robots.txt, where a shared group of AI agents contradicted my * rules, and wrote it up in robots.txt for AI crawlers: the user agents that matter. The same logic applies to every token on this page.
What I could not verify
I did not test whether these bots honour the rules in practice. Anthropic’s article says they do. I have not seen logs from my own site to confirm or contradict that, and I would be inventing a finding if I told you otherwise.
If you want to test it on your own site, the cleanest approach is a robots.txt change, then a log search a few days later. The Perplexity side of the same question, where the vendor itself says one fetcher generally ignores robots.txt, is in PerplexityBot and Perplexity-User.
Questions people ask
What is the ClaudeBot user agent?
ClaudeBot is Anthropic’s name for its training-data crawler, and the robots.txt token is ClaudeBot. The help article I read does not print the full user agent string, so match on the name.
Does ClaudeBot respect robots.txt?
Anthropic states that its bots honour industry-standard robots.txt directives and support Crawl-delay.
Is blocking ClaudeBot enough to stay out of Claude?
No. ClaudeBot relates to training data. Claude-User and Claude-SearchBot are separate, and Anthropic describes disabling them as reducing visibility in user-directed search and search results.
Should I block by IP address?
Anthropic advises against relying on that for an opt-out, because blocking IPs can stop it reading your robots.txt.
Turning the three bots into a policy
Treat it as three questions, one per bot.
Should your future content be excluded from model training? That is ClaudeBot. Should Claude be able to fetch your page when someone asks about it? That is Claude-User, and Anthropic says disabling it may reduce your visibility for user-directed search. Should your content be indexed to improve Claude’s search results? That is Claude-SearchBot, with the same warning about reduced visibility.
A site that wants to stay findable but out of training sets answers no, yes, yes. A site that wants nothing to do with Anthropic answers no, no, no. The file has to name all three tokens either way, because blocking one does not imply the others. Remember that each subdomain needs its own robots.txt.
Do this today
Search your robots.txt for ClaudeBot, Claude-User and Claude-SearchBot. For each one, write down whether you want it allowed. If only the old anthropic-ai or Claude-Web lines are there, add the three current names with the policy you chose.
Resources
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, read 11 October 2026.
- RFC 9309, Robots Exclusion Protocol.
- OpenAI: Overview of OpenAI crawlers, read 11 October 2026, for the user agent comparison.
- Perplexity: crawler documentation, read 11 October 2026, for the user agent comparison.