How to Block AI Crawlers in WordPress with robots.txt

On this page, 8 sections

To block AI crawlers in WordPress, add a separate robots.txt group for each crawler’s token, each with Disallow: /. For example, User-agent: GPTBot followed by Disallow: /. That is a request, not a lock: the standard itself says its rules are not a form of access authorization. Compliant crawlers follow it. Anything else needs a firewall.

The part that trips people up on WordPress is where robots.txt actually comes from. WordPress can generate the file on the fly, but a physical file on disk wins over it. I checked my own server on 11 October 2026 and found exactly that, so I start there before the vendor tokens.

Below: which file your site really serves, a tested snippet for the virtual file, what each vendor documents about its tokens, and the one group-merging trap that can quietly undo a block. Every vendor claim links to that vendor’s own page, read on 11 October 2026.

Find out which robots.txt your site really serves

WordPress has a function, do_robots(), that builds a virtual robots.txt when a request for /robots.txt reaches WordPress. I read its source on my server on 11 October 2026.

Its default output is three lines: User-agent: *, Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. It runs the do_robotstxt action first and then passes the text through the robots_txt filter, which receives the output and a flag for whether the site is public.

But if a real robots.txt file sits in the web root, the web server hands that file out and WordPress never runs. On my server there is one.

I compared the bytes: the file on disk and the file the live site returns have the same MD5 hash (ea4597ff3a5775a813e408ea0d7c6023, 1,954 bytes). So on this site, a robots_txt filter would do nothing at all.

You can run the same check in two commands. The first fetches what the world sees; the second looks for a physical file in your site’s root directory (replace the path with yours).

curl -s https://example.com/robots.txt | md5sum
md5sum /path/to/site-root/robots.txt

On macOS, use md5 where I wrote md5sum. If the second command says “No such file”, your site uses the virtual file, and the snippet in the next section is the right tool. If the hashes match, edit the physical file instead.

SEO plugins and some plugins that manage AI-crawler rules write the physical file too, so look for their marker comments before you edit by hand. My file has a block that begins with a BEGIN RankReady comment, and anything inside a block like that gets rewritten by the plugin that owns it.

The tested snippet for the virtual robots.txt

This must-use style snippet appends one Disallow group per token to whatever WordPress generates. It returns the text unchanged when the site is set to discourage search engines, so it does not add noise to a private site. Put it in a small plugin file or in wp-content/mu-plugins/.

<?php
/*
 * Plugin Name: Block AI training crawlers in robots.txt
 */
add_filter(
	'robots_txt',
	function ( $output, $is_public ) {
		if ( ! $is_public ) {
			return $output;
		}

		$agents = array( 'GPTBot', 'ClaudeBot', 'CCBot', 'Google-Extended' );

		foreach ( $agents as $agent ) {
			$output .= "\nUser-agent: {$agent}\nDisallow: /\n";
		}

		return $output;
	},
	20,
	2
);

I checked this two ways on 11 October 2026. php -l on PHP 8.5.10 printed “No syntax errors detected”. Then I ran it against a stub that mimics add_filter and feeds it WordPress’s default three lines.

With $is_public set to '1' it appended four groups: GPTBot, ClaudeBot, CCBot and Google-Extended. With '0' it returned the default text untouched. I did not activate it on a live WordPress install, because my live site serves a physical file.

I then fed the resulting groups to Python’s urllib.robotparser. GPTBot and ClaudeBot came back not allowed to fetch a post URL; OAI-SearchBot and Googlebot came back allowed, because they match only the * group, which does not block them. Matching ignores case: gptbot was also blocked, which agrees with RFC 9309.

What each vendor says its token does

Blocking “AI crawlers” is not one decision. Most vendors run separate bots for training, for search and for fetches a person triggers. Here is what each documents, read on 11 October 2026.

Token	What the vendor’s page says	If you disallow it
GPTBot (OpenAI)	Crawls content that may be used to train its generative AI foundation models.	OpenAI says this indicates the content should not be used for training.
OAI-SearchBot (OpenAI)	Used to surface sites in ChatGPT’s search features.	OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links. It can take about 24 hours for a robots.txt change to apply.
ChatGPT-User (OpenAI)	Visits a page when a user asks ChatGPT or a custom GPT a question.	OpenAI says that because the actions are user-initiated, robots.txt rules may not apply.
ClaudeBot (Anthropic)	Collects web content that could contribute to training.	Anthropic says it signals that your future material should be excluded from training datasets.
Claude-User, Claude-SearchBot (Anthropic)	User-directed retrieval, and search-quality indexing.	Anthropic says disabling either may reduce your visibility in user search results.
PerplexityBot, Perplexity-User	PerplexityBot surfaces and links sites in Perplexity search; Perplexity-User supports user actions.	Perplexity says neither is used to crawl content for AI foundation models. Disallowing PerplexityBot therefore targets search listing, not training.
Google-Extended (Google)	A standalone token that manages whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps.	Google says it has no separate HTTP user agent string: crawling uses existing Google user agents, and the token acts only as a control.
CCBot (Common Crawl)	Common Crawl’s crawler.	Common Crawl documents `User-agent: CCBot` with `Disallow: /` to opt out.

Two consequences follow. First, a “block all AI” list that includes OAI-SearchBot, Claude-SearchBot or PerplexityBot is also a “stay out of AI search results” list. Decide which you want.

Second, the user-triggered agents are the weak point of any robots.txt approach: OpenAI says the rules may not apply to ChatGPT-User, and Google documents that its user-triggered fetchers ignore robots.txt. I covered the Google-Extended details in Google-Extended is not a user agent.

The shared-group trap

Many robots.txt files, mine included, list a long run of crawler names stacked above one set of rules. Mine stacks about thirty AI and search agents over Allow: /, plus allow lines for /llms.txt, /llms-full.txt and markdown files, and its Content-Signal line currently reads ai-train=yes, search=yes, ai-input=yes.

If I wanted to block GPTBot by simply adding a new User-agent: GPTBot group at the bottom, would it work?

RFC 9309 says that when more than one group matches a crawler, the rules are combined into one group. Then, when an Allow and a Disallow rule are equivalent, the Allow rule should be used. Allow: / and Disallow: / are equivalent. So a crawler listed in both groups is, under the standard, allowed.

I tested a stripped-down version of the pattern in Python’s urllib.robotparser: GPTBot came back allowed. That parser takes the first matching group rather than merging, so it reaches the same answer by a different route, which is the real lesson: parsers differ, and an ambiguous file is not a block.

The fix is boring. Remove the token from the shared group, then give it its own group. Keep each token in exactly one place, and put a comment above the group saying why.

Why robots.txt is not enough on its own

RFC 9309 states that these rules are not a form of access authorization. A crawler that ignores them gets the page anyway. Two other limits matter in practice:

Caching. The RFC says crawlers should not use a cached copy of robots.txt for more than 24 hours unless the file is unreachable. A block is not instant.
Spoofed names. Anyone can send a crawler’s name in the User-Agent header. In 25 days of my own access logs, about one in five requests under twelve claimed AI and search crawler names went to probe paths, which points to a scanner using borrowed names. I am writing the full counts up in a separate access-log post.

Do not block by IP to enforce a robots.txt opt-out. Anthropic’s support page says that blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because doing so impedes its ability to read your robots.txt. Its pages say its bots honour robots.txt directives and support the non-standard Crawl-delay line.

If you need enforcement, the layer is a firewall rule at your CDN or server, ideally one that checks the crawler’s published IP ranges as well as its name. Perplexity’s documentation shows WAF allow rules that combine User-Agent and IP conditions.

If you are on Cloudflare, its documentation describes a “Block AI bots” setting and per-behaviour AI bot policies under Security settings. The page, last updated 1 July 2026, also describes changes to defaults and a deprecation scheduled for 15 September 2026, written in the future tense. I did not test any Cloudflare setting for this post, so check what your dashboard shows today.

Questions people ask

Will blocking GPTBot remove me from ChatGPT answers?

OpenAI documents each setting as independent. It describes GPTBot as the training crawler and OAI-SearchBot as the one for ChatGPT search, and its own example is allowing OAI-SearchBot to appear in search results while disallowing GPTBot. Blocking only GPTBot is the documented way to opt out of training and stay eligible for search.

Does blocking Google-Extended affect Google Search?

Google describes Google-Extended as a standalone token for Gemini training and grounding, separate from Search crawling. The same page says Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal.

Do I need a robots.txt plugin?

Not for four groups. A plugin earns its place when you want a settings screen or automatic updates to the token list. Whatever you use, run the md5 check above first so you know whether you are editing the file your visitors’ crawlers receive.

Do this today

Run curl -s https://yoursite.com/robots.txt and read it top to bottom. Check whether a physical file exists. Pick which of training, search and user-triggered fetches you want to block, and write one group per token. Then fetch the file again and confirm the groups appear once each.

For the full token reference, see robots.txt for AI crawlers, and for checking what actually arrives, reading WordPress access logs.

Resources

OpenAI: crawlers and user agents (GPTBot, OAI-SearchBot, ChatGPT-User), read 11 October 2026.
Anthropic: how site owners can block its crawlers, read 11 October 2026.
Perplexity: crawlers and WAF configuration, read 11 October 2026.
Google: common crawlers, including the Google-Extended token, read 11 October 2026.
Common Crawl: CCBot, read 11 October 2026.
RFC 9309, Robots Exclusion Protocol.
WordPress: do_robots() and the robots_txt filter, read 11 October 2026.
Cloudflare: Block AI Bots, read 11 October 2026, last updated 1 July 2026.

How to Block AI Crawlers in WordPress (robots.txt Is Not a Lock)