If your website traffic reports have dropped while your hosting bills have climbed, AI crawlers are often part of the explanation. These are automated programs from companies like OpenAI, Anthropic, and Perplexity that read your site to train AI models, power AI search, or answer a question someone asked a chatbot. They show up in your server records but rarely in Google Analytics, so the two sources your team reports from now tell different stories. At Curious Minds Media, we’ve spent the past year reconciling those records on client sites, and the decisions that follow affect both hosting cost and whether your company appears in AI answers.
Why your server logs and GA4 tell different stories
Your server keeps a record of every request made to your website, whether it comes from a person or a program. Google Analytics only counts a visit when the visitor’s browser runs its tracking script, and most AI crawlers never run it. The result is a growing volume of activity your servers handle and pay for that never appears in the traffic reports leadership reviews.
The reason is how these crawlers read pages. In a December 2024 analysis, Vercel and MERJ found that none of the major AI crawlers they tested ran a page’s scripts, while Google’s and Apple’s crawlers do. That has a second consequence for sites built on modern JavaScript frameworks or headless setups: if key content only appears after scripts run, these crawlers see a mostly empty page, and that content can’t be cited in AI answers. During technical audits, we check priority pages for exactly this.
The four kinds of AI traffic, in plain terms
AI companies publish documentation on their crawlers, and each one has a specific job. That job decides whether the crawler ever sends a customer back to you.
Training crawlers from companies such as OpenAI, Anthropic, Meta, and Common Crawl collect content to build AI models. They read widely and send almost no visitors back, because a trained model has no link to your page.
AI search crawlers build the indexes behind ChatGPT search, Claude, Perplexity, and Microsoft’s Copilot, which can draw on Bing’s index. These work much like Google’s crawler, and they’re the ones that make it possible for your company to be cited, and clicked, in an AI answer.
On-demand fetchers visit a specific page because a person asked an AI assistant to read it. OpenAI and Perplexity both say their on-demand fetchers may not follow a site’s crawler rules, since a person initiated the request, while Anthropic says its fetcher does follow them.
AI browsers, such as OpenAI’s ChatGPT Atlas and Perplexity’s Comet, act on a user’s behalf inside a full web browser. Google Analytics can’t tell them apart from people, so they show up in reports as ordinary visitors, sometimes without any record of where they came from. Blocking them usually means blocking the customer who sent them, which is a poor trade for any business that sells online.
What this costs, and how it skews reporting
The cost shows up first on the hosting side. Crawlers read pages people seldom visit, such as old archives and filtered listings, and those requests tend to bypass the caching that keeps a site fast and inexpensive to run. The same Vercel and MERJ analysis found that AI crawlers request missing pages and redirects far more often than Google does, which usually points to old links left over from a past site migration. A complete redirect plan during a migration reduces that waste. Cloudflare’s published data on how often each AI company crawls compared with how often it sends visitors back puts the training-focused operators far above traditional search engines, and that matches what we see on client sites.
The reporting problems are quieter and can matter more. In one analytics audit for a publishing client, retired websites that still carried the company’s tracking code were sending a steady stream of visits to error pages, and a cluster of high-volume traffic from a handful of countries showed almost no engagement. Neither problem was visible in the top-line report, and both were shaping the editorial decisions the client made from it.
Some of the change never reaches your servers at all. A Pew Research Center analysis of browsing data from 900 U.S. adults in March 2025 found that Google users clicked a traditional search result in 8% of visits when an AI summary appeared, compared with 15% when none did. On one travel client account in 2026, sessions declined while the client reported its conversion rate was higher than it had ever been, because the visitors who did arrive from AI platforms were further along in their decision. Lower traffic alongside better conversions reads as a problem only when the two numbers are reviewed separately.
The decisions that belong with leadership
Each AI company runs separate crawlers for separate jobs, and your site can treat them differently. A site can stay visible in ChatGPT search while keeping its content out of OpenAI’s model training, because OpenAI controls the two through separate settings. That makes access a business decision before it becomes a technical one. Whether your content should help train AI models is a policy call for leadership, while blocking the crawlers behind AI search usually costs visibility you want to keep.
Enforcement carries the technical risk, because a site’s crawler rules are only a request and not every crawler honors them. The dependable controls live at the hosting layer, where traffic can be slowed or filtered before it reaches the site. On managed hosting built using Cloudflare Enterprise, such as the BigScoots configurations we manage, filtering happens at the network edge. One mistake we’ve seen is a security rule strict enough to turn away the crawlers that make AI citations possible. We’ve run into the same failure with our own tools: in fall 2026, security settings on one client site blocked every request from our maintenance platform, and a caching plugin on another site rejected the same requests.
Our general approach is to allow AI search crawlers and on-demand fetchers, treat training crawlers as a decision for the client, and limit anything that can’t be verified.
See: AI Crawlers Are Increasing: How to Manage Bot Traffic and Use Rate-Limiting Safely.
How we report AI traffic to leadership
When leadership sees sessions decline year over year, the instinct is to call it an SEO failure, and that reading is often incomplete when AI summaries are answering questions that used to produce a click. We report human visits and AI-referred visits as separate figures, and we pair both with conversion rate and lead quality so the conversation stays on outcomes.
Since May 2026, Google Analytics has grouped visits from recognized AI assistants such as ChatGPT, Gemini, and Claude into their own channel, which makes that split easier to show, although visits from AI Overviews still count as regular Google search. When hosting costs come up, we add crawler volume from the server records so the conversation has numbers behind it.
Questions to ask your team or technical partner
A short set of questions will tell you whether your site is handling this well. Ask whether anyone compares your server records with Google Analytics on a regular schedule, and whether your reports separate human visits from AI-referred visits. Ask what your site currently allows each type of AI crawler to do, and who made that decision. Ask whether your security and rate-limit settings have been checked against the crawlers behind AI search, so the site isn’t turning them away by accident. If those answers aren’t clear, that’s the place to start.
For your technical team
The crawler names to look for in server logs are GPTBot, OAI-SearchBot, and ChatGPT-User for OpenAI; ClaudeBot, Claude-SearchBot, and Claude-User for Anthropic; and PerplexityBot and Perplexity-User for Perplexity.
This robots.txt policy stays eligible for ChatGPT search, opts out of OpenAI model training, and slows Anthropic’s training crawler:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Crawl-delay: 1
Anthropic supports Crawl-delay, but Google ignores it and OpenAI doesn’t document support for it. Verify crawlers against the IP ranges OpenAI, Anthropic, and Perplexity publish, and use reverse DNS for Google, since user-agent strings are easy to spoof. Repeated 403 or 429 responses to OAI-SearchBot or Claude-SearchBot usually mean a WAF rule or rate limit is blocking a crawler the site wants, and OpenAI states that sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers.
Google-Extended and Applebot-Extended are robots.txt control tokens that never appear in logs, and Google-Extended has no effect on AI Overviews. GA4 excludes known bots automatically and its data filters only cover internal and developer traffic, so cleanup relies on removing tags from retired domains, checking hostnames, and building segments that isolate near-zero-engagement sessions.
Working with Curious Minds Media
Curious Minds Media has built and maintained WordPress sites since 2004. Our in-house, US-based engineers handle hosting, security, and analytics within the same engagement, so one team reconciles your server records with your reports and makes the access decisions above with your business goals in view. If your traffic numbers and hosting costs have started telling different stories, our SEO and AIO optimization team can help work out which traffic is which.
