Every day, AI crawlers scan your website. They read your content, pull the text, and feed it into language models and answer engines. Most site owners have no idea this is happening. They cannot tell you which bots showed up, how often, or which pages got taken. The reason is simple: the analytics tool on their site was never built to see a crawler. If you are watching Google Analytics and nothing else, you are blind to the traffic that now decides whether AI engines know your site exists.
This guide covers how AI crawler tracking actually works, why your current analytics miss it, and how to read the data once you can finally see it.
Why your analytics cannot see AI crawlers
Google Analytics and most popular trackers work the same way. They run a snippet of JavaScript in the visitor's browser. When the page loads, the script fires, an event gets recorded, and you see a visit. That model assumes a browser is on the other end.
AI crawlers are not browsers. They request the raw HTML of a page, parse it on their own servers, and move on. They do not wait for JavaScript to run. A joint analysis by Vercel and MERJ tracked more than 500 million GPTBot fetches and found no evidence of JavaScript execution at all. ChatGPT, Claude, and Perplexity crawlers behave the same way: fetch the HTML, take what they need, never trigger your tracking pixel.
So the bot visit happens, your server hands over the content, and your analytics dashboard stays empty. The crawler might as well be invisible. That is the gap, and it is silent. Your reports look normal while AI companies quietly read everything you publish.
How AI crawler tracking works
The fix is to stop tracking in the browser and start reading the server. Every time any bot requests a page, your web server records the request: the user-agent string the bot sent, the exact page, the timestamp, and the response code it got back. That record exists whether or not JavaScript ever runs.
Each crawler announces itself with a user-agent string. GPTBot sends one that contains "GPTBot." ClaudeBot sends one with "ClaudeBot." Meta's crawler identifies as Meta-ExternalAgent. By reading those strings out of the server's own log of requests, you can see every page a given bot touched and when it touched it. This is the most reliable method available because it captures the raw request at the network level, before any rendering step that a crawler would skip anyway.
One caveat: not every bot tells the truth. robots.txt is a voluntary standard, not a law, and while most reputable crawlers respect it, some ignore it and a few disguise their user-agent entirely. That is exactly why you need the real request record rather than a policy file. The log shows you what happened. robots.txt only states what you asked for.
Which AI crawlers actually matter
There are thousands of known bots, but only a handful drive most of the AI traffic. As of 2026, Googlebot still leads overall crawl volume. Meta-ExternalAgent sits in second place, and GPTBot in third, ahead of crawlers like Bytespider, ClaudeBot, and Bingbot. The names worth watching on your own logs include:
- GPTBot - OpenAI's training crawler, pulling content for model training.
- ClaudeBot - Anthropic's crawler doing the same.
- Meta-ExternalAgent - Meta's high-volume training crawler.
- OAI-SearchBot - OpenAI's crawler for answering searches inside ChatGPT.
- PerplexityBot - fetches pages live to answer a user's question in the moment.
- Applebot, Amazonbot, GoogleOther - platform crawlers feeding their own AI features.
A raw list of bot names is a start, but it does not tell you what any of them want from you. That is the part most tracking gets wrong.
Names are not enough - sort every bot by purpose
Seeing that GPTBot hit your site forty times last week is interesting. Knowing what those forty hits mean for your business is the actual job. A crawler taking your content to train a model is doing something completely different from a crawler fetching you live to answer a question, and lumping them together teaches you nothing.
citAEOtion reads your server-level crawler activity and sorts every bot into four categories:
- AI Training - bots pulling your content into model training pipelines, like GPTBot, ClaudeBot, and Meta-ExternalAgent.
- AI Search - bots indexing you so AI engines can surface you in their search results.
- AI Assistant - bots fetching you live to answer a user's question right now, like PerplexityBot.
- Data Scraper - everything else taking your content, attribution optional.
Once the bots are sorted, the numbers start to mean something. A training crawler is deciding whether your content shapes the model. A search crawler is deciding whether you get cited in an answer. A scraper is just taking. Separate them and you can see your real position: who showed up, what they took, and when.
Training comes first, citations follow
Here is the pattern that a sorted feed reveals and a raw hit count never will. The training bots move first. When a model trains on your content, the search and assistant citations come later. The scale is not small: on Vercel's network, GPTBot alone made roughly 569 million requests in a single month and ClaudeBot another 370 million. Those are training-weighted crawlers working the open web for fresh content right now.
So in your own crawl mix, the training hits are the leading indicator. Feed the training bots clean, open, well-structured content today and you are setting up citations months out. We have watched it on our own sites: open the gates to AI training instead of blocking it, the crawler volume climbs, and the citations follow. You only see that cause and effect if you are measuring the actual bots, which is the whole point of tracking them by purpose rather than counting them in a pile.
What to do once you can see the data
Tracking is not the goal. The decisions it unlocks are. Once you can see which bots hit which pages, you can act on it:
- Decide allow, throttle, or block - per bot. If a scraper hits your pricing page every hour and sends nothing back, you have a case to block it. If PerplexityBot fetches you live and surfaces you in answers, you almost certainly want to keep it. You cannot make that call from a single on/off switch. You make it from the record of what each bot actually does.
- Catch the cost. Heavy crawlers eat bandwidth and server capacity, and the most aggressive spread requests across wide IP ranges to dodge rate limits. The log tells you which bot is burning the most, so you target the throttle instead of swinging a blunt rule that takes out real users too.
- Find your blind spots. If your most important pages never get crawled by the engines you care about, that is a signal to fix structure or framing so they do.
The danger of blocking blind is real. Cut off a training or assistant crawler without knowing what it does and you can quietly remove yourself from the answers that engine generates. Data first, then the decision.
Become the answer, measured on evidence
The most useful thing about a real crawler feed is the loop it opens. Your bot mix tells you whether you are being cited or merely consumed. That tells you whether your framing is landing. That tells you what to change. Then you watch the search and assistant hits climb to prove the change worked. Measure, learn, reframe, repeat, on evidence instead of vibes.
That is the citAEOtion thesis in one line: the GA of AI. Full data. No BS. It is a WordPress plugin with a roughly five-minute install. It names every known AI crawler, sorts each one by purpose, shows page-level hit counts, and tracks the trend over time as new bots appear. See how it works, or start tracking your real crawler traffic.
Frequently Asked Questions
Why does Google Analytics miss AI crawler traffic?
Google Analytics relies on JavaScript to record a visit. AI crawlers fetch the raw HTML of a page and do not run JavaScript, so the tracking code never fires and the bot request never appears in your reports. Server-log tracking captures the request at the network level instead, which is why it sees crawlers that browser-based analytics cannot.
How do I identify which AI bot visited my site?
Each crawler sends a user-agent string that names it - GPTBot, ClaudeBot, Meta-ExternalAgent, PerplexityBot, and so on. Your server records that string with every request, alongside the page and timestamp. Reading those records tells you exactly which bot hit which page and when, without relying on the bot to cooperate beyond announcing itself.
Can I block AI crawlers completely?
You can block many of them with robots.txt or server-level rules, but robots.txt is voluntary and not every bot honors it. Some ignore it and a few disguise their user-agent. That is why log-based tracking matters: it confirms whether your blocks are actually working and shows you the bots that slip past.
Do AI crawlers hurt site performance?
High volumes of any bot traffic consume server resources, and the heaviest AI crawlers can add up. For most sites it is not a serious problem until the volume gets large, but you only know which bot is responsible if you are reading your logs. That lets you throttle the offender precisely instead of blocking everything.
What is the difference between an AI training bot and an AI search bot?
A training bot, like GPTBot or ClaudeBot, pulls your content to help build or update a model. A search bot indexes you so an AI engine can surface you in its answers. Sorting the two apart matters because training crawls tend to come first and the citations follow, so training activity is the early signal that you are on track to be cited.