Claude-User, Claude-SearchBot and ClaudeBot: Anthropic's Crawlers Explained

Claude-User is the agent Anthropic uses to fetch a web page when a person asks Claude a question that needs that page. It is separate from ClaudeBot, which collects content for model training, and from Claude-SearchBot, which indexes content for search. Anthropic says all three respect robots.txt, so you can allow or block each one on its own.

Quick answer: Anthropic documents three web agents. Claude-User fetches pages in response to individual user questions, and blocking it stops Claude from retrieving your content for those queries. Claude-SearchBot indexes content to improve Claude's search answers, and blocking it stops that indexing. ClaudeBot collects content that may be used for training, and blocking it tells Anthropic to exclude your future content from training datasets. Each agent has its own robots.txt token, and Anthropic publishes the IP addresses its bots use so you can verify visits.

Why "Claude-User" confuses people

Search your server logs for "Claude-User" and it is easy to assume a person opened your page. It is not a person's browser. It is Anthropic's software fetching the page because a person asked Claude something. The human never sees your site directly. Claude reads it on their behalf.

That makes Claude-User one of the most useful lines in your logs. It is not a background crawl. Each request is tied to a live question, which is about as close as you can get to seeing demand for your content inside an AI assistant.

What Anthropic says about each agent

Anthropic's crawler documentation is the source to trust here. Summaries on other sites go stale, and agent names do change.

Claude-User

Anthropic says Claude-User "supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent." It also says Claude-User lets site owners control which sites can be accessed through these user-initiated requests.

On blocking, the documentation is direct: disabling Claude-User "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search."

Claude-SearchBot

Claude-SearchBot is the search indexer. Anthropic says it analyzes online content "specifically to enhance the relevance and accuracy of search responses." Disabling it "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results."

Put simply, Claude-SearchBot builds the background knowledge of what your pages contain, while Claude-User fetches a page at the moment someone needs it. Both feed answers. Neither is described as a training crawler.

ClaudeBot, briefly

ClaudeBot is the training crawler. Anthropic says it collects web content "that could potentially contribute to" model training, and that restricting it "signals that the site's future materials should be excluded from our AI model training datasets." We cover it in depth in our ClaudeBot guide, so here the key point is the split: blocking ClaudeBot is a training decision, not a visibility decision.

Claude-User, Claude-SearchBot and ClaudeBot compared

This three-way split, training, search indexing, and user-triggered fetching, is now common across AI companies. Our guide to AI crawler categories shows how other operators' agents map to the same groups.

Does Claude-User respect robots.txt?

According to Anthropic, yes. The documentation states that Anthropic's bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." That covers Claude-User as well as the crawlers.

This is worth knowing because not every operator handles user-triggered fetches the same way. Some treat a user-requested fetch like a person clicking a link and do not apply robots.txt to it. With Anthropic, a robots.txt rule for Claude-User is the documented control.

Anthropic also supports the non-standard Crawl-delay directive, which lets you slow down how often its bots request pages. Its documentation gives this example:

User-agent: ClaudeBot Crawl-delay: 1

Robots.txt rules for each agent

Because robots.txt works by user agent token, you can treat each Anthropic agent differently. A common setup keeps pages eligible for Claude's answers while opting out of training:

User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: ClaudeBot Disallow: /

Whether that is right for you depends on how you want your work used. The point is that the choice is yours per agent. For the tradeoffs, see our guide on whether to allow or block AI bots, and for syntax details, robots.txt for AI crawlers.

One caution: Anthropic's documentation notes that blocking by IP address alone may not work correctly or persistently, because it can stop the bots from reading your robots.txt. Use robots.txt as the primary control.

How to tell which Claude agent hit your page

1. Open your raw access log rather than a JavaScript analytics report. Bots rarely run tracking scripts, so client-side tools miss them.
2. Read the user agent field for the exact token: Claude-User, Claude-SearchBot, or ClaudeBot.
3. Check the requesting IP against the list Anthropic publishes at claude.com/crawling/bots.json. A matching name with a non-matching IP is likely a spoofed request.
4. Look at the pattern. Claude-User requests track what people are asking, so they tend to land on individual pages rather than sweeping the whole site.
5. Sort by status code. A Claude-User request that gets a 404 or 403 is a question your site failed to help answer.

What Claude-User data tells you about your content

Claude-User requests are a demand signal. The pages they hit are the pages people's questions are pulling into Claude's answers. That makes them a practical shortlist for updates: refresh the facts, tighten the opening answer, and make sure the page loads cleanly.

Watch the gaps too. If Claude-SearchBot visits a section regularly but Claude-User never does, those pages are indexed but may not match what people ask. If Claude-User keeps hitting a page that returns an error, you are losing answers you already earned.

Set a monthly habit. Pull Anthropic agent hits from your logs, compare them with the previous month, and look for new pages, changes in volume, and recurring errors. Small regular checks beat one large audit you never repeat.

If you would rather see Claude-User, Claude-SearchBot, and ClaudeBot requests per page and per status code without reading raw logs, citAEOtion is a $99 Lifetime Deal, and you can book a 20-minute walkthrough to see it on a real site.

Frequently Asked Questions

Is Claude-User a real person visiting my site?

No. Claude-User is Anthropic's agent fetching a page because a person asked Claude a question. The person sees Claude's answer, not your site. Blocking the Claude-User token stops those fetches but does not block any person from visiting you directly.

Does Claude-User respect robots.txt?

Anthropic says its bots honor industry standard robots.txt directives, and that applies to Claude-User. Anthropic's documentation says Claude-User lets site owners control which sites can be accessed through user-initiated requests. A rule targeting the Claude-User token is the documented way to allow or block it.

What is the difference between Claude-User and Claude-SearchBot?

Claude-SearchBot indexes content to improve Claude's search results in the background. Claude-User fetches a specific page at the moment a user's question needs it. Anthropic says blocking either one may reduce your site's visibility in search.

Can I allow Claude-User but block ClaudeBot?

Yes. Robots.txt rules are written per user agent token, so you can allow Claude-User and Claude-SearchBot while disallowing ClaudeBot. Anthropic says restricting ClaudeBot signals that your future content should be excluded from training datasets.

How do I verify that a Claude-User request is genuine?

Match the requesting IP address against the list Anthropic publishes at claude.com/crawling/bots.json. A request that carries the Claude-User name but comes from an address outside that list should be treated as an unknown bot, whatever its user agent says.