TLDR: AI traffic intelligence reads real crawler data to spot your content winners early, because a page an AI assistant bot fetches again and again is one already surfacing in live answers. AI traffic intelligence means using real crawler data to predict which of your pages will win in AI answers. Sort AI bots into […]
A user-agent string is trivially faked. Verify GPTBot, ClaudeBot, and PerplexityBot by IP range and forward-confirmed reverse DNS. Exact steps inside.
PerplexityBot is Perplexity's search crawler that indexes public pages so they can be surfaced and linked in Perplexity answers, and it identifies itself as PerplexityBot/1.0 in your server logs. PerplexityBot is Perplexity's crawler for search, not for model training. It sends the user agent PerplexityBot/1.0, reads server-delivered HTML, and indexes pages so Perplexity can cite […]
A good WordPress AI crawler tracking plugin reads real server traffic, not JavaScript. Here is what separates a log-based collector from analytics that miss every bot.
ClaudeBot is Anthropic's web crawler that collects public content to train the models behind Claude, and it identifies itself as ClaudeBot/1.0 in your server logs. ClaudeBot is Anthropic's training crawler. It sends the user agent ClaudeBot/1.0, reads the HTML your server returns, and feeds that text into the data used to train Claude. Anthropic also […]
GPTBot is OpenAI's web crawler that collects public page content to train its AI models, and it identifies itself as GPTBot/1.4 in your raw server logs. GPTBot is OpenAI's training crawler. It sends the user agent GPTBot/1.4, fetches raw HTML from public pages, and adds that text to the data pool used to train the […]
Prompt-based AI visibility samples what AI models say; log-based AI visibility counts what AI crawlers actually did on your site. Prompt-based vs log-based AI visibility is the core split in AEO tools. Prompt-based tools like Profound, Peec AI, Otterly, and Scrunch ask AI models sample questions and count brand mentions, which is sampling. Log-based tools […]
Google Analytics cannot show which AI bots crawl your site. Server logs can. Here is how to read them and what a log-based dashboard reveals.
Google-Extended is a robots.txt control token that decides whether your Google-crawled content can be used for Gemini AI, and it is not a separate crawler with its own user agent. Google-Extended is a robots.txt token, not a crawler. It has no user agent and fetches nothing. It only tells Google whether content already crawled by […]
Automated traffic now exceeds human traffic, AI crawlers make up nearly 80% of AI bot activity, and GPTBot leads AI-only crawlers. Every figure cited to Cloudflare, Fastly, and Imperva.