Launch Deal$99 Lifetime Deal (limited time)View Offer

Category: AI Visibility Engineering

PerplexityBot Explained: User Agents, Verification, and the robots.txt Controversy

PerplexityBot is Perplexity's search crawler that indexes public pages so they can be surfaced and linked in Perplexity answers, and it identifies itself as PerplexityBot/1.0 in your server logs. PerplexityBot is Perplexity's crawler for search, not for model training. It sends the user agent PerplexityBot/1.0, reads server-delivered HTML, and indexes pages so Perplexity can cite […]

What Makes a Source Get Cited by AI? Lessons from Crawler Data in 2026

TLDR: A source gets cited by AI when it has strong brand search demand, quantitative claims, answer-first structure, and 40 to 60 word paragraphs, with most citations pulled from the top third of the page. A source gets cited by AI when it pairs strong brand search demand with quantitative claims, answer-first structure, and short […]

ClaudeBot Explained: Anthropic's Crawlers, User Agents, and robots.txt Control

ClaudeBot is Anthropic's web crawler that collects public content to train the models behind Claude, and it identifies itself as ClaudeBot/1.0 in your server logs. ClaudeBot is Anthropic's training crawler. It sends the user agent ClaudeBot/1.0, reads the HTML your server returns, and feeds that text into the data used to train Claude. Anthropic also […]