Everything I
have checked.
- PerplexityBot User Agent: What It Is and How to Allow It PerplexityBot is Perplexity's search crawler; Perplexity-User generally ignores robots.txt. The strings, IP lists and firewall rules, read 11 October 2026.
- ClaudeBot, Claude-User, Claude-SearchBot: What Each Crawler Does Anthropic documents three crawlers: ClaudeBot for training, Claude-User for user requests, Claude-SearchBot for search. What each does and how to control it.
- 31,305 Impressions, 81 Clicks: What My Search Console Shows Fifteen pages hold 56.5% of my 31,305 impressions, but most clicks came from the other 344. What 90 days of Search Console data shows, and what…
- Which AI Crawlers Visit My Site: 25 Days of Real Access Logs I counted 12 AI and search crawler names in 25 days of nginx logs. About one in five of their requests hit probe paths, which points…
- GPTBot vs OAI-SearchBot vs ChatGPT-User: What Each One Does GPTBot trains, OAI-SearchBot feeds ChatGPT search, ChatGPT-User fetches on request. The user agents, robots.txt rules and IP lists, read 11 October 2026.
- Google-Extended User Agent: It Is a Token, Not a Crawler Google-Extended has no user agent string, so it never shows in your logs. What the token controls for Gemini, what it leaves alone in Search, and…
- How to Find All URLs on a Website: 4 Methods and What Each Misses Sitemap, crawler, WordPress REST API or site: search. On my own site the sitemap first listed 400 of 779 published posts. Here is how to find…
- WordPress Sitemaps: Core Versus Plugin, and Whether AI Crawlers Read Them Yoast and Rank Math disable core sitemaps with the same one-line filter. What core emits, the 2,000 URL default, and what AI crawler docs say.
- Canonical Tags on WordPress: What Breaks Them, and How to Audit Yours Core emits rel=canonical on singular pages only, and every SEO plugin removes that function anyway. Traced through core, Yoast and Rank Math source.
- Structured Data and AI Answers: What Schema Is Actually Doing for You Now Google says schema is not required for generative AI search. What it does document, what practitioners assume, and how to audit your own graph.
- robots.txt for AI Crawlers: The Actual User-Agent Strings, and What Each One Honours GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended and Bingbot, with the real strings and the RFC rule that breaks most AI blocks.
- How to Measure Whether AI Search Sends You Anything Three partial signals and no complete one. Server logs, the new Search Console generative AI report, Bing AI Performance, and each one's limits.