robots.txt for AI crawlers uses User-agent and Disallow lines to allow or block bots like GPTBot and ClaudeBot, but the file is voluntary, so some crawlers ignore it entirely.
robots.txt for AI crawlers is a plain-text file at your site root that names each AI bot in a User-agent line and uses Disallow to control what it may fetch. GPTBot, ClaudeBot, Google-Extended, and CCBot honor it; some crawlers such as Bytespider have a history of ignoring it.
Quick answer: To block an AI crawler, add a block with its exact user-agent token and "Disallow: /". To allow it, use "Disallow:" with nothing after it, or omit the bot. OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Common Crawl's CCBot, and Google-Extended obey robots.txt. User-initiated fetchers like ChatGPT-User may not apply robots.txt, and bots such as Bytespider have been reported to ignore it. robots.txt is a public request, not a firewall, so pair it with server or edge rules when a block must be enforced.
How robots.txt works for AI bots
robots.txt is a single file at https://yourdomain.com/robots.txt. Crawlers that follow the Robots Exclusion Protocol read it before fetching pages. Each rule block starts with one or more User-agent lines that name a bot, followed by Allow and Disallow paths. A bot picks the most specific block that matches its token and follows those rules. If no block matches, it falls back to the block for "*". A related file, the llms.txt proposal, suggests preferred pages but does not allow or block any crawler.
Two facts matter before you edit anything. First, matching is by user-agent token, and the tokens are case sensitive in practice, so spelling counts. Second, compliance is voluntary. A well-behaved crawler obeys robots.txt; a badly-behaved one reads it and does whatever it wants, or skips it.
Exact syntax to block the major AI crawlers
This block denies the main training and search crawlers from OpenAI, Anthropic, Common Crawl, and Google's AI training program, plus Amazon and ByteDance:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / Each block is independent. "Disallow: /" means the entire site is off limits to that named bot. A blank line separates blocks. Do not merge unrelated bots under one User-agent line unless you want identical rules for all of them.
Exact syntax to allow AI crawlers
Allowing is the default. If a bot is not named and there is no restrictive "*" block, it may crawl. To state permission explicitly, use an empty Disallow:
User-agent: GPTBot Disallow: User-agent: ClaudeBot Disallow: An empty Disallow value means "nothing is disallowed," which permits full access. If you want to allow a bot everywhere except one folder, combine the two:
User-agent: GPTBot Allow: / Disallow: /private/ What each token controls
GPTBot is OpenAI's crawler for training foundation models. OAI-SearchBot powers ChatGPT search results. ChatGPT-User fetches a page when a user asks ChatGPT to visit it. ClaudeBot is Anthropic's web crawler. PerplexityBot is Perplexity's declared crawler. Google-Extended is not a crawler at all; it is a robots.txt token that controls whether your content trains Gemini and Vertex AI, and blocking it does not affect Google Search indexing. CCBot is Common Crawl, whose archive feeds many AI datasets. Amazonbot and Bytespider serve Amazon and ByteDance respectively.
The bots that ignore robots.txt
Here is the part vendors gloss over. robots.txt is an honor system. OpenAI, Anthropic, Google, and Common Crawl document that their listed crawlers respect it. But user-initiated fetches through ChatGPT-User may not apply robots.txt because a person requested that specific page. Bytespider has repeatedly been reported crawling sites that disallowed it. And any operator can spoof a user agent or run an undeclared bot that never appears in your rules.
So a Disallow line stops the polite crawlers and does nothing to the rest. If you need a hard block, enforce it at the server or CDN with rules that check the user agent and IP, or use a service that challenges unverified bots. robots.txt sets policy; your server enforces it.
You cannot manage what you cannot see
Before you block or allow anything, find out which bots actually hit you and what they take. Writing "Disallow: /" for nine crawlers is guesswork if you have never confirmed that even three of them visit your site. citAEOtion reads real WordPress server traffic and classifies every AI and search crawler by actual hits, so you see which bots requested which pages, how often, and with which HTTP status. That turns robots.txt from a copy-paste ritual into a decision backed by data. See how citAEOtion classifies crawler traffic, or check the Single Site plan at $34.99/mo on the pricing page. Running this for clients? The agency plans add white-label reports.
The protocol itself is defined in the official specification: IETF RFC 9309, the Robots Exclusion Protocol.
Frequently Asked Questions
Where does robots.txt go?
At the site root, so it loads at https://yourdomain.com/robots.txt. Rules in a subfolder file are ignored.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only governs AI training use. Googlebot still crawls and indexes your pages for Search when Google-Extended is disallowed.
Is robots.txt enough to stop AI scraping?
No. It stops compliant bots only. For enforcement, add server or CDN rules that block by user agent and verified IP.
How do I block every AI bot at once?
Name each token in its own block with "Disallow: /". A single "*" block also affects normal search engines, so prefer explicit AI tokens if you still want to appear in Google.
How do I know my robots.txt is working?
Check server logs for requests from each user agent after a change. citAEOtion shows those hits and their status codes without manual log parsing.