AI Crawler & Bot Traffic 2026: The Data

AI crawler and bot traffic statistics crossed a structural threshold in 2026: on June 3, Cloudflare CEO Matthew Prince shared Radar data showing that automated requests now make up 57.5% of HTML traffic to web content, against 42.5% from humans — the first time machines have held the majority in internet history. This page is the sourced reference behind that headline and the economics underneath it.

What matters for publishers and SEOs is not the single 57.5% figure; it is the asymmetry behind it. Most AI crawling exists to harvest training data, and training crawlers return almost no referral traffic. ClaudeBot, by Cloudflare’s own measurement, crawled tens of thousands of pages for every visit it sent back. That crawl-without-credit gap is the through-line of every number below.

This is the data page — a citable reference with every number traced to the publisher’s own page and a date. For the mechanics of which crawler to allow or block and how, read the companion AI crawler access-control decision matrix. For the rights and news angle on where this training data comes from, see our breakdown of the Common Crawl training-data showdown. Definitions and methodologies differ by source — we flag the measurement basket on every figure that needs one.

The single most-cited number of 2026 came from a June 3 post by Cloudflare co-founder and CEO Matthew Prince, sharing Cloudflare Radar data: bots now generate 57.5% of HTML web traffic, with humans at 42.5%. Prince had forecast this crossover at SXSW but expected it roughly a year later — it arrived ahead of schedule. The measurement covers HTML HTTP requests across Cloudflare’s network and explicitly excludes video streaming, email, and gaming traffic, so it is a measure of crawlable web content, not all internet packets.

A second, independently-reported figure tells a compatible story with a different ruler. Imperva’s Bad Bot Report 2026 put automated traffic at 53% of all web traffic in 2025, up from 51% in 2024. The two numbers measure different baskets — Cloudflare counts HTML requests; Imperva counts all web traffic including app and API calls — so they should not be averaged or treated as the same metric. What they share is direction: across independent vendors, machines now move more of the web than people do.

Not all bots are AI bots. Within verified bot traffic, Cloudflare put AI crawlers at 20.3% in May 2026, with AI-search bots adding a further 6.5% — making AI-related activity about 26.7% of verified bot traffic. The rest is the long-standing population of search-engine indexers, uptime monitors, feed fetchers, and the like. Among the named crawlers, Googlebot remained the single largest at 27.26% of AI-adjacent bot HTTP requests in May 2026, ahead of GPTBot and ClaudeBot.

The OpenAI-versus-Anthropic race is closer than headlines suggest and it flips month to month. In May 2026 Cloudflare put GPTBot at 11.48% of AI bot HTTP requests and ClaudeBot at 9.73% — a reversal from April 2026, when ClaudeBot led 11.69% to 9.84%. Reading any single month as a durable ranking is a mistake; the volumes are volatile. Year over year the trend is clearer: per Cloudflare’s August 2025 analysis, GPTBot’s share of verified bot traffic rose from 4.7% in July 2024 to 11.7% in July 2025, and ClaudeBot grew from 6% to nearly 10% over the same window.

The most counterintuitive finding in the data refutes a common assumption — that AI bots mostly power AI search and therefore mostly return citations and clicks. They do not. Cloudflare attributed 51.8% of AI crawler requests to training in May 2026, with mixed-purpose (training plus retrieval) adding another 35.7%, and search-only purpose accounting for just 9.3% — though that search slice is rising, up from 7.5% in April 2026. In Cloudflare’s full-year 2025 data, training drove nearly 80% of AI bot activity, up from about 72% the year before.

This is the analytical crux of the whole subject. When more than half of AI crawling exists to build training corpora rather than to answer a live user query, the majority of that activity has no mechanism to send a visitor back to the source. Your content is consumed to improve a model that may later answer questions without ever linking to you. That is a different economic relationship than the search-engine bargain SEOs have operated under for two decades, and it is why the zero-click dynamic matters more, not less, in the AI era. We unpack that shift in our analysis of the 2026 zero-click search landscape.

The crawl-to-refer ratio — pages crawled per referral visit sent back — is the single clearest measure of how fair the exchange is, and the spread across platforms is staggering. The figures below come from SEOmator’s March 2026 GEO Data Report reading Cloudflare Radar data, and they carry a hard caveat: ratios are tied to a specific time window and must never be averaged across windows. ClaudeBot, for example, measured 23,951:1 over January–March 2026 and improved to 11,122:1 for the single week of May 25 to June 1, 2026 — both real, neither universal.

One measurement limitation cuts in publishers’ favor and is worth stating plainly: Cloudflare notes that native-app traffic — the Claude and ChatGPT mobile and desktop apps — often arrives without a Referer header, which can make these ratios overstate the imbalance. The direction is not in doubt, but the exact multiples should be read as order-of-magnitude, not precise.

The pattern is unmistakable: platforms that run a consumer search product — Google, DuckDuckGo, and to a lesser degree Bing-backed Copilot — return traffic in roughly the volume they take, because a crawl exists to power a result the user clicks. Pure training and answer crawlers do not, because the model answers in place. Anthropic sits at the extreme end of that spectrum, while Perplexity, which surfaces sources inline, is one to two orders of magnitude more referral-efficient than the worst training crawlers. For the log-level evidence behind how these crawlers actually move through a site, see our 30-day site-log study of AI crawler behavior.

The blocking response has been large and fast. After Cloudflare made AI-crawler blocking the default for new domains on July 1, 2025, its customers blocked 416 billion AI bot scraping requests in the five months through early December 2025, per data Matthew Prince shared with Search Engine Land. More than 2.5 million sites had chosen to fully disallow AI training as of August 2025, and over a million Cloudflare customers had activated AI-crawler blocking.

In robots.txt directives specifically, GPTBot is the most-blocked AI crawler, appearing in 5.52% of DISALLOW rules in Q1 2026 per TechnologyChecker.io’s analysis, ahead of CCBot (5.08%), ClaudeBot (4.88%), Google-Extended (4.44%), and Bytespider (4.23%). There is a notable gap among news publishers: while 79% block at least one AI training bot, only 46% block Google-Extended, Google’s AI training crawler — a reluctance Matthew Prince has linked to Google’s decision to merge its search and AI crawlers into a single user-agent.

That combined-crawler design is the most underreported structural advantage in the data. Cloudflare’s figures imply Google’s combined search-plus-AI crawler gives it roughly 3.2× more webpage access than OpenAI, 4.6× more than Microsoft, and about 4.8× more than Anthropic or Meta — because a publisher who blocks Google’s AI training also forfeits Google search visibility, a trade few will make. For the precise directives that map each crawler to allow, block, or AI-Overviews-only, our complete robots.txt and meta-robots reference is the practical companion, and Google now exposes opt-out controls inside Search Console’s AI performance settings.

One control that gets more attention than it earns is llms.txt. Adoption stood at about 10.13% of nearly 300,000 domains surveyed in 2026 — fairly uniform across traffic tiers — yet analysis of hundreds of millions of LLM bot traffic events found that the AI bots driving search visibility rarely fetch the file. Implementing llms.txt is low-cost and harmless, but the current data does not support treating it as a lever that changes AI search visibility. We cover what it is and isn’t in our llms.txt specification and markdown-first content guide.

The crawl-to-refer story is bleak, but the referrals that do arrive are unusually valuable — at least in retail, where the best data exists. Adobe’s Q2 2026 AI Traffic Report, drawing on Adobe Analytics across its US retailer clients, found AI-referred traffic to US retailers grew 393% year over year in Q1 2026, peaking at 1,151% YoY growth in December 2025. Those AI-sourced visitors converted 42% better than non-AI channels, generated 37% more revenue per visit, and spent 48% more time on-site as of March 2026.

Two cautions keep this honest. First, the Adobe figures are retail-specific — other industries Adobe measured grew far less (media and tech in the double digits, not the triple), so the 393% number does not generalize to all websites. Second, AI referral volume is still small relative to traditional search even as it grows fast; websearchapi.ai’s read of Cloudflare Radar put AI search visits up 42.8% YoY, from 15.6 billion to 27.4 billion between Q1 2025 and Q1 2026 — real growth, off a base that remains a fraction of conventional search.

The machine-majority web did not arrive overnight. The numbers below trace a roughly two-year arc from the first AI-crawl surge to the human-traffic crossover, each anchored to a dated, primary-sourced milestone.

Read together, these numbers point to a forward shift that is already underway. As training and answer crawling keep outpacing search crawling, the share of your content that is consumed without any referral mechanism will keep rising — which means the old proxy of crawl budget as a leading indicator of traffic is breaking down. A page can be crawled tens of thousands of times and earn a single visit. Crawl volume and referral value have decoupled, and they are unlikely to recouple on their own.

The practical projection for the next several quarters is a two-track strategy. On one track, decide deliberately which crawlers to allow, block, or permit only for AI-search — a content-licensing and access-control decision, not a default. On the other, instrument AI referral sources properly so that when the high-quality traffic Adobe documented in retail shows up in your vertical, you can see it, attribute it, and credit AI-assisted journeys rather than only last-click AI visits. Neither track is optional once machines move the majority of the web — both are at the core of how we run our agentic SEO engagements.