# How to Block AI Crawlers, and Whether You Should

https://citaeotion.ai/how-to-block-ai-crawlers/

You can block AI crawlers with robots.txt, server rules, or Cloudflare, but whether you should depends on data about which bots hit you and whether AI answers send you traffic or just take content.
To block AI crawlers, name each bot in robots.txt with Disallow, add server rules that reject their user agents, or turn on AI bot blocking in Cloudflare. Whether you should block depends on whether a given bot sends referral traffic or only scrapes your content.
Quick answer: There are three layers to block AI crawlers. robots.txt stops compliant bots like GPTBot and ClaudeBot by policy. Server rules in Apache or Nginx reject requests by user agent and enforce the block. Cloudflare offers a one-click toggle to block AI bots at the edge. Blocking makes sense when a crawler takes content and sends no traffic back, when you sell content, or when crawl volume strains your server. Allowing makes sense when AI search tools cite you and drive clicks. Decide per bot, using real hit data, not a blanket rule.
The three layers where you can block AI crawlers
Blocking is not one switch. It is three layers, each stronger and more work than the last.
Layer 1: robots.txt (policy)
The lightest option. Name each bot and disallow it:
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

Compliant crawlers obey this. It is a public request, not enforcement, so a bot that ignores robots.txt walks right through. For the full token list and allow syntax, see the robots.txt for AI crawlers guide.
Layer 2: server rules (enforcement)
To make a block real, reject the request at the server. In Nginx:
if ($http_user_agent ~* "(GPTBot|ClaudeBot|PerplexityBot|CCBot|Bytespider)") {
    return 403;
}

In Apache with mod_rewrite inside .htaccess:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|PerplexityBot|CCBot|Bytespider) [NC]
RewriteRule .* - [F,L]

This returns 403 to any request whose user agent matches, whether or not the bot reads robots.txt. A determined operator can still spoof a browser user agent, so this stops honest identification, not every possible fetch.
Layer 3: Cloudflare (edge)
If your site sits behind Cloudflare, you can block known AI bots at the network edge before requests reach your origin. Cloudflare AI Crawl Control, formerly AI Audit, lets you allow or block individual crawlers, manage robots.txt on your behalf, and see which AI services requested your content. Edge blocking also uses verification signals, so it catches more spoofing than a user-agent string alone. This is the strongest of the three and works across your whole site with little configuration.
Whether you should block: the case for each side
Blocking is a business decision, not a default. Both sides are real.
Reasons to block. A training crawler like CCBot or GPTBot can ingest your content and return nothing. If you sell access to writing, research, or data, letting it be scraped for free undercuts the product. High-frequency crawlers can also add server load and bandwidth cost. And some site owners simply do not consent to their work training commercial models.
Reasons to allow. AI search and assistant tools increasingly cite sources and link out. If PerplexityBot or OAI-SearchBot indexes you and ChatGPT or Perplexity then names you in an answer, that can send qualified visitors and build authority. Block those bots and you disappear from the surfaces where buyers now ask questions. Blocking training crawlers while allowing search crawlers is a common middle path.
A decision framework
Decide per bot, not for the whole category. Ask three questions for each crawler that actually hits you:
1. Does this bot drive referral traffic or citations back to me? If yes, lean allow.
2. Does this bot only ingest content with no return, or hit me hard enough to cost money? If yes, lean block.
3. Do I sell the content this bot is taking? If yes, block and enforce at the server or edge.
A workable default for many sites: allow search-oriented crawlers such as OAI-SearchBot and PerplexityBot, block pure training and dataset crawlers such as CCBot and GPTBot, and enforce anything sensitive at the edge.
Every one of these questions needs data you probably do not have
You cannot answer "does this bot drive traffic" or "does this bot hit me hard" from a robots.txt file. You need to know which bots reach your site, which pages they take, how often, and with what status codes. Most site owners are guessing.
citAEOtion reads real WordPress server traffic and classifies every AI and search crawler by actual hits, so the block-or-allow decision rests on evidence instead of headlines. You see GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Amazonbot, and Bytespider by page, frequency, category, and HTTP status. Prompt-based tools guess by sampling model answers; citAEOtion measures the requests. See how the tracking works, compare plans from $34.99/mo, or explore agency plans with white-label reports if you make these calls for clients.
OpenAI documents its crawler tokens and their robots.txt behavior in the official reference: OpenAI bots documentation.
Frequently Asked Questions
Does robots.txt actually stop AI crawlers?
It stops the compliant ones. Bots that ignore robots.txt, or spoof a browser, need server or Cloudflare enforcement.
Will blocking AI crawlers hurt my SEO?
Blocking Google-Extended does not affect Google Search. Blocking AI search crawlers can remove you from AI answers, which is a separate visibility channel from blue-link search.
Should I block every AI bot?
Rarely. A blanket block also cuts off bots that cite and link to you. Decide per bot using real hit data.
What is the strongest way to block a bot?
Edge blocking through Cloudflare, backed by server rules, is stronger than robots.txt because it enforces rather than requests.
How do I confirm a block is working?
Watch for the bot in your traffic after the change. citAEOtion shows whether a crawler still reaches you and what status code it receives.
See which AI bots are hitting you before you block, in a live demo

---
Site: citAEOtion - https://citaeotion.ai/
LLM index: https://citaeotion.ai/llms.txt | Full text: https://citaeotion.ai/llms-full.txt