SEO
Technical and on-page SEO checked against crawl data and Search Console, not against what the industry repeats to itself.
- Google Scaled Content Abuse Policy: What It Says and Doesn’t What Google's scaled content abuse policy and its generative AI guidance actually say, what they leave undocumented (a publishing rate), and a five-question self-audit.
- Does llms.txt Work? What AI Vendors Actually Say About It Three AI vendors' crawler pages never mention fetching llms.txt, and Google says you need no AI text files. What I checked on 11 October 2026, and…
- Do You Need an XML Sitemap? Mine Was Missing 163 Posts Google says a small, well-linked site may skip a sitemap. My index hid 163 of 563 posts for over a month. What Google ignores, and a…
- PerplexityBot User Agent: What It Is and How to Allow It PerplexityBot is Perplexity's search crawler; Perplexity-User generally ignores robots.txt. The strings, IP lists and firewall rules, read 11 October 2026.
- ClaudeBot, Claude-User, Claude-SearchBot: What Each Crawler Does Anthropic documents three crawlers: ClaudeBot for training, Claude-User for user requests, Claude-SearchBot for search. What each does and how to control it.
- 31,305 Impressions, 81 Clicks: What My Search Console Shows Fifteen pages hold 56.5% of my 31,305 impressions, but most clicks came from the other 344. What 90 days of Search Console data shows, and what…
- Which AI Crawlers Visit My Site: 25 Days of Real Access Logs I counted 12 AI and search crawler names in 25 days of nginx logs. About one in five of their requests hit probe paths, which points…
- GPTBot vs OAI-SearchBot vs ChatGPT-User: What Each One Does GPTBot trains, OAI-SearchBot feeds ChatGPT search, ChatGPT-User fetches on request. The user agents, robots.txt rules and IP lists, read 11 October 2026.
- Google-Extended User Agent: It Is a Token, Not a Crawler Google-Extended has no user agent string, so it never shows in your logs. What the token controls for Gemini, what it leaves alone in Search, and…
- How to Find All URLs on a Website: 4 Methods and What Each Misses Sitemap, crawler, WordPress REST API or site: search. On my own site the sitemap first listed 400 of 779 published posts. Here is how to find…