robots.txt for AI crawlers uses User-agent and Disallow lines to allow or block bots like GPTBot and ClaudeBot, but the file is voluntary, so some crawlers ignore it entirely. robots.txt for AI crawlers is a plain-text file at your site root that names each AI bot in a User-agent line and uses Disallow to control […]
llms.txt is a proposed plain-text file that lists your most important pages in Markdown so AI models can find clean content fast, but adoption stays limited and no major AI company has confirmed its crawlers read it. llms.txt is a plain-text file you place at the root of your website that points AI language models […]
TLDR: An AI visibility audit finds the gap between what Google ranks and what AI cites by reading real crawler data, so you fix the exact pages AI bots ignore instead of guessing. An AI visibility audit maps where your brand actually stands in the gap between ranking on Google and getting cited by AI. […]
In 40 days, AI crawlers hit 20 sites 114,644 times. The surprise: most of it is training harvest, not the search crawling that gets you cited.
AI Overviews and AI Mode cite pages that answer a specific sub-question, prove real expertise, and stay crawlable for Googlebot. The 2026 playbook plus how to verify.
TLDR: Every AI platform has its own citation fingerprint, ChatGPT leans on Wikipedia, Perplexity on Reddit, Claude on authoritative sources, so tailor content by platform and track which bots actually crawl your pages. AI citation patterns differ by platform: ChatGPT leans on Wikipedia almost half the time, Perplexity leans on Reddit, and Claude wants authoritative, […]
A user-agent string is trivially faked. Verify GPTBot, ClaudeBot, and PerplexityBot by IP range and forward-confirmed reverse DNS. Exact steps inside.
PerplexityBot is Perplexity's search crawler that indexes public pages so they can be surfaced and linked in Perplexity answers, and it identifies itself as PerplexityBot/1.0 in your server logs. PerplexityBot is Perplexity's crawler for search, not for model training. It sends the user agent PerplexityBot/1.0, reads server-delivered HTML, and indexes pages so Perplexity can cite […]
A good WordPress AI crawler tracking plugin reads real server traffic, not JavaScript. Here is what separates a log-based collector from analytics that miss every bot.
ClaudeBot is Anthropic's web crawler that collects public content to train the models behind Claude, and it identifies itself as ClaudeBot/1.0 in your server logs. ClaudeBot is Anthropic's training crawler. It sends the user agent ClaudeBot/1.0, reads the HTML your server returns, and feeds that text into the data used to train Claude. Anthropic also […]
GPTBot is OpenAI's web crawler that collects public page content to train its AI models, and it identifies itself as GPTBot/1.4 in your raw server logs. GPTBot is OpenAI's training crawler. It sends the user agent GPTBot/1.4, fetches raw HTML from public pages, and adds that text to the data pool used to train the […]
Prompt-based AI visibility samples what AI models say; log-based AI visibility counts what AI crawlers actually did on your site. Prompt-based vs log-based AI visibility is the core split in AEO tools. Prompt-based tools like Profound, Peec AI, Otterly, and Scrunch ask AI models sample questions and count brand mentions, which is sampling. Log-based tools […]
Google Analytics cannot show which AI bots crawl your site. Server logs can. Here is how to read them and what a log-based dashboard reveals.
Automated traffic now exceeds human traffic, AI crawlers make up nearly 80% of AI bot activity, and GPTBot leads AI-only crawlers. Every figure cited to Cloudflare, Fastly, and Imperva.
You cannot improve what you cannot measure. Prompt sampling is noisy; server-log crawler data is ground truth. What to track, and how citAEOtion reports it.