Is a high AI crawl-to-referral ratio bad for my site?

Not by itself. A high ratio driven by training crawlers is expected, because training crawls are not designed to send visits. Look first at the ratio for search and user-triggered agents. Read it as a ceiling, because apps often strip the referrer, and check whether the same platforms cite you. A high ratio on those agents, sustained over several equal windows, is the case worth acting on.

Why do published crawl-to-referral ratios differ between AI platforms?

The likely reason is the request mix. Training crawls are not designed to send visits. Perplexity documents that Perplexity-User may visit a page to help answer a question and include a link to the page in its response. A platform whose crawling serves training more than linked answers will tend to show a higher ratio. Treat published values as directional network aggregates.

Does blocking AI training crawlers increase referral traffic?

We know of no published study that measures this. Blocking a training crawler such as GPTBot removes requests from the numerator, so your overall ratio falls, but that does not create referrals. Search and user-triggered agents are the ones that fetch pages for answers. To judge the effect on your site, compare per-agent ratios and referral counts over equal windows before and after the change.

How do I count ChatGPT app visits that arrive without a referrer?

You cannot count them directly, only estimate them. Identify landing pages that receive direct traffic and that, as far as you can tell, only AI answers link to, such as deep pages rarely shared elsewhere. Treat their unexplained direct visits as possible AI referrals, report a low and a high estimate next to the measured referrals, and never merge the estimate into the measured count.

Should Googlebot be included when I calculate the ratio?

Keep Google separate. Google states that sites appearing in AI features such as AI Overviews and AI Mode are included in the overall search traffic in Search Console, so you cannot isolate referrals that came from AI features alone. A Googlebot ratio therefore mixes classic search and AI features. Report it on its own line rather than comparing it with OpenAI, Anthropic or Perplexity.

What a Good AI Scrape-to-Referral Ratio Is, and How to Measure It

There is no universal good AI scrape-to-referral ratio. It counts a platform's crawler HTML requests per referral visit from that platform. Published values are network-wide aggregates, and your number only means something once training, search and user-triggered bots are separated and visits without a referrer are accounted for.

What the AI scrape-to-referral ratio measures, and how it is calculated

Cloudflare divides a platform's crawler requests that returned Content-type: text/html by the HTML requests whose Referer header carries that platform's hostname (methodology consulted 14 November 2026). The unit is crawler requests per referral.

Crawler requests, referral sessions and citations are distinct signals; the ratio combines requests and referrals and says nothing about whether an answer named you (see how an AI mention differs from an AI citation).

AI referral counts are often small in absolute terms, so a few visits swing the ratio. They also understate exposure: native apps and copied URLs often send no Referer, so the observed ratio sits above the true value.

With no referrals, the ratio is undefined; report it as undefined, never as an invented figure. Hypothetical example for October: verified OAI-SearchBot and ChatGPT-User fetch N HTML pages, and R visits carry ChatGPT's hostname in the Referer; the ratio is N divided by R.

Published crawl-to-referral ratios by AI platform, compared

Cloudflare's Radar Year in Review reports that OpenAI's ratio "reached as much as 3,700:1 in March"; we restate only that figure, verified verbatim on 14 November 2026.

Platform	Crawler types counted	Published ratio	Period covered	Source date
OpenAI	Platform-associated user agents	As much as 3,700:1	March, peak value	Consulted 14 Nov 2026
Anthropic, Perplexity	Same method	Published, not restated here	See report	Consulted 14 Nov 2026
Google	AI features not separable	Not isolable	Not applicable	Consulted 14 Nov 2026

We expect values to depend on the request mix: training crawls are not designed to send visits; user-triggered fetches can be. Perplexity documents that Perplexity-User may visit a page "and include a link to the page in its response" (consulted 14 November 2026).

Google is not comparable: "sites appearing in AI features (such as AI Overviews and AI Mode) are included in the overall search traffic in Search Console" (Google, consulted 14 November 2026), so an AI-only Google ratio cannot be isolated. Published ratios are network-wide aggregates that move over time; treat them as direction, not a target.

How to compute your own ratio from server or CDN logs

Keep requests, referrals and citations as separate counts, and put only HTML page requests over a stated window in the numerator.

Verify crawler identity by user agent and published IP ranges

User agents are easy to fake. Keep a request only when its user agent and source IP both match the vendor's documentation; OpenAI and Perplexity publish IP range files (checked 14 November 2026).

Separate training, search and user-triggered requests

OpenAI documents GPTBot for training, OAI-SearchBot for search and ChatGPT-User for "certain user actions". Anthropic lists ClaudeBot, Claude-SearchBot and Claude-User; Perplexity separates PerplexityBot from Perplexity-User (all consulted 14 November 2026). Compute each class's ratio per platform. Excluding training can change the verdict: a high overall ratio may hide a moderate search and user ratio.

Count referrals and correct for missing referrers

Count HTML requests whose Referer carries each platform's hostname; for sessions, see setting up an AI referral channel in Google Analytics. To estimate what the header misses, count direct visits to landing pages that only AI answers appear to link to, and report them as an estimated range, never as measured referrals.

What counts as a good ratio for your site

Judge the search and user-triggered ratio first, since those fetches serve answers that can link to you. Read the observed figure as a ceiling: referrals are undercounted, so the true ratio is lower. With small referral counts, swings and percentage changes are mostly noise; wait for several equal windows before reading a trend.

We expect content type to matter, as an inference, not a benchmark: documentation may earn verification clicks, while news may be summarised in full. Complement clicks with a scrape-to-citation view: crawler requests per tracked citation. We know of no reference values for it.

What to do when the ratio is high: robots.txt, rate limits and pay-per-crawl

In robots.txt, blocking a training crawler keeps your pages out of what that agent collects; blocking a search or user-triggered agent can keep them out of cited answers (see which AI crawlers to allow or block in robots.txt). Other options are rate limits at the CDN, direct licensing, or Cloudflare pay per crawl, under which crawlers receive "an HTTP 402 Payment Required response with pricing" (consulted 14 November 2026). We know of no published study measuring their effect on referrals, so compare equal windows before and after a single change.

Where to start this week: compute your baseline ratio

Export a month of raw HTML logs.
Keep verified bots; split training, search and user-triggered.
Count referrals per platform hostname.
Compute and date each ratio, writing "undefined" where referrals are absent, plus an estimated missing-referral range.
Re-run on an equal window regularly.

To see the citation side of the ratio, run a free Namedrop scan and check whether ChatGPT, Perplexity, Gemini and Claude cite your site.