ClaudeBot is Anthropic's web crawler that collects public content to train the models behind Claude, and it identifies itself as ClaudeBot/1.0 in your server logs.
ClaudeBot is Anthropic's training crawler. It sends the user agent ClaudeBot/1.0, reads the HTML your server returns, and feeds that text into the data used to train Claude. Anthropic also runs Claude-User for live user fetches and Claude-SearchBot for search indexing. You control all three with robots.txt rules that name each user agent, and because none of them run JavaScript, only a server-log tool records their visits.
Quick answer: Anthropic runs three named agents. ClaudeBot (user agent ClaudeBot/1.0) crawls the open web for model training. Claude-User (Claude-User/1.0) fetches a page when a person asks Claude about it during a conversation. Claude-SearchBot (Claude-SearchBot/1.0) indexes pages to improve Claude's search-style answers. Each token is separate, so a rule aimed at ClaudeBot does not touch Claude-User or Claude-SearchBot. Anthropic honors standard robots.txt directives, including Crawl-delay. All three read only server-delivered HTML and skip JavaScript, so Google Analytics never records them and your web server access log is the single source of truth.
What ClaudeBot is and what it does
ClaudeBot is the crawler Anthropic uses to gather publicly available web content for training its generative AI models. When ClaudeBot requests a page, it downloads the HTML and keeps the text. Anthropic describes this as collecting web content to help improve the capability and safety of its models. Pages you allow can be used in training. Pages you disallow are excluded from that dataset per Anthropic's stated policy.
ClaudeBot is a training crawler, not the agent that answers a live question inside Claude. A ClaudeBot line in your logs means Anthropic read your content for training, not that a user just asked Claude about your page. Two other agents handle live and search activity, and both are covered below. ClaudeBot is one of the major AI crawlers reading the open web, and it behaves much like OpenAI's GPTBot and Google-Extended.
The exact Claude user agent strings
Anthropic publishes a fixed token for each agent. At the time of writing the full strings are:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)For filtering and robots.txt, match on the product tokens ClaudeBot, Claude-User, and Claude-SearchBot. The contact address in each string points back to Anthropic, which is a quick sanity check when you read a log line.
ClaudeBot vs Claude-User vs Claude-SearchBot
Anthropic separates its web activity into three jobs. Treat each as its own decision.
ClaudeBot/1.0 -> model training Claude-User/1.0 -> live fetch when a person asks Claude Claude-SearchBot/1.0 -> indexing for Claude's search answersClaudeBot is the bulk training crawler. Disallowing it keeps your content out of Claude's training data.
Claude-User fires when a human asks Claude about a specific page or task and Claude fetches that URL to answer. Because a person triggered it, Anthropic treats it as a user action rather than automated crawling.
Claude-SearchBot gathers and indexes pages so Claude can return better search-style results with citations. Blocking it can reduce how often your pages surface in Claude's answers, so weigh that against your goals before you disallow it.
Anthropic split these tokens apart on purpose. The result is that opting out of training is a different switch from opting out of live fetches or search visibility. You can allow the agents that send you referral value and block only the one you object to.
How to control Claude bots in robots.txt
Anthropic follows standard robots.txt allow, disallow, and Crawl-delay directives. Place your rules at the root of your domain, for example https://example.com/robots.txt.
To block only training while keeping search and live fetches open:
User-agent: ClaudeBot Disallow: /To block every Claude agent across the whole site:
User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Disallow: /To slow ClaudeBot down instead of blocking it, add a Crawl-delay:
User-agent: ClaudeBot Crawl-delay: 10To keep training out of one area while allowing the rest:
User-agent: ClaudeBot Disallow: /private/Because the tokens are independent, a block on ClaudeBot alone still lets Claude-SearchBot index you and still lets Claude-User fetch a page a person asked about. Decide each line based on whether you want training out, search out, or both.
How to track ClaudeBot on your site
ClaudeBot does not execute JavaScript. It reads the HTML your server sends and stops there. Google Analytics and other browser-tag tools only fire when their script runs in a real browser, which ClaudeBot never does. That means your GA reports show zero ClaudeBot activity even during heavy crawling. The requests are real. The tracking tag simply never runs for a bot.
The one place a ClaudeBot request is recorded is your web server access log. That is the data citAEOtion reads to classify AI crawlers on WordPress. It reads real traffic, tags each hit by its true user agent, and sorts ClaudeBot, Claude-User, and Claude-SearchBot into categories like AI Training, AI Assistant, and AI Search. You can see every Claude bot hit on your site, which pages each one pulled, how often, and the HTTP status it received, instead of assuming Anthropic has or has not read a page.
A single site runs $34.99 a month, with agency plans and a Founders Club discount available. See the tiers on the citAEOtion pricing page, or book a live demo to watch Claude traffic classified on a live WordPress dashboard.
Anthropic's own crawler reference, with the current tokens and blocking guidance, is in Anthropic's official crawler documentation.
Frequently Asked Questions
Is ClaudeBot the same as Claude?
No. ClaudeBot is the crawler that gathers training data. Claude is the assistant people chat with. When a live Claude session fetches a page, that request comes from Claude-User or Claude-SearchBot, not ClaudeBot.
Does blocking ClaudeBot remove me from Claude's answers?
No. Blocking ClaudeBot keeps your content out of training. Whether your pages appear in Claude's search-style answers is governed by Claude-SearchBot. To stay visible while staying out of training, allow Claude-SearchBot and disallow ClaudeBot.
What is the difference between Claude-User and ClaudeBot?
ClaudeBot is automated bulk crawling for training. Claude-User is a single fetch triggered by a person asking Claude about a specific page. One is a background data collector; the other is a user action.
Why does Google Analytics never show ClaudeBot?
ClaudeBot does not run JavaScript, and Google Analytics only counts visitors whose browsers execute its script. ClaudeBot never runs the script, so GA cannot see it. A server-log tool like citAEOtion sees it because it reads the raw request.
Does ClaudeBot respect Crawl-delay?
Yes. Anthropic honors the Crawl-delay directive in robots.txt, so you can slow ClaudeBot without blocking it outright.