Agentic Crawler Behavior: 30-Day Site Log Study 2026

Most of what gets written about AI crawlers is downstream of vendor blog posts and rumor. The actual behavior — how often GPTBot revisits, where ClaudeBot prefers to go, what PerplexityBot does when nobody is asking it anything — lives in your server access logs. So we pulled the logs.

Twelve production sites participated in this study: four B2B SaaS properties, three ecommerce stores, three agency websites, and two publishers. Site sizes ranged from 380 indexed pages to roughly 48,000. The window was thirty consecutive days from late March to late April 2026. We tracked eleven canonical AI user-agents, normalized for prefetch and back-fill bursts, and compared the traffic against the corresponding Googlebot baseline on each site.

The findings are consistent enough across verticals that they should change how SEO and platform teams plan crawl-budget, content-refresh, and edge-cache strategy in the second half of 2026. The single biggest surprise is not the volume — it is the divergence in behavior between bots that all look similar from the outside. GPTBot, ClaudeBot, and PerplexityBot crawl your site for fundamentally different reasons, on different cadences, with different path preferences. That divergence is what this post documents.

Search Console, Bing Webmaster, and the Yandex equivalent all report the legacy crawler footprint clearly. None of them report on AI crawlers. The vendors themselves publish little or no usage data, and the third-party panels that exist are too small and too US-east-coast-publisher-skewed to translate to production agency workloads.

That leaves server access logs — the one ground-truth source that sees every user-agent string, every path, every status code, every byte transferred. They are also the source most teams have stopped paying attention to. Modern logging stacks (Vercel, Cloudflare, Datadog, Axiom) keep the data; few SEO teams query it. The thesis here is simple: if you want to plan for agentic search in 2026, the access log is the tool that already tells you what matters.

The sample frame was selected for vertical mix and infrastructure mix, not for traffic size. Four B2B SaaS sites, three ecommerce stores, three agency websites, and two publishers participated. Indexed-page counts ranged from 380 (a niche SaaS landing site) to roughly 48,000 (a mid-size publisher). All twelve sites are real production properties; none are honeypots.

Identification ran on the user-agent string with secondary verification by reverse DNS for GPTBot, ClaudeBot, Google-Extended, and PerplexityBot (the four bots that publish verifiable IP ranges). User-triggered fetches were separated from scheduled crawls by user-agent (ChatGPT-User and Perplexity-Userare explicit) and corroborated by request-rate signature. Bot bursts during a single back-fill were normalized into per-day medians to avoid skewing one site's numbers based on a one-time event.

The volume picture is the headline finding. GPTBot is the most aggressive single bot on every site we measured. ClaudeBot is a distant second on B2B SaaS and agency sites; on ecommerce, the ranking flips and Bytespider becomes the highest-volume bot of all (with very aggressive product-page coverage). PerplexityBot looks small in the median but is the most variable — quiet for days, then spiking on a viral query.

A few notes on what the chart hides. The Bytespider line is the median across all twelve sites; on the three ecommerce stores the number is closer to 6,500/day, with intense product-page coverage. The Google-Extended line is steady and small because Google-Extended explicitly opts into training only; the full AI footprint from Google reaches you through Googlebot and the AI Overviews path. And the PerplexityBot median understates the spike risk — the upper-decile minute saw 240 requests per minute when a viral query referenced one of the publisher sites.

The behavior split is sharper than the volume split. Volume tells you who is showing up; behavior tells you why. The four major bots cluster cleanly into four crawl personalities, each with distinct implications for what you should publish, what you should cache, and what you should write into your robots.txt and llms.txt files.

The cost story depends almost entirely on architecture. On statically-exported Next.js or Astro sites behind Vercel, Cloudflare, or Fastly edge caches, AI crawlers add zero meaningful CPU cost — every fetch hits the cache. On dynamically-rendered sites with no edge cache (a large fraction of the WordPress, Magento, and custom-PHP installed base), the same crawl traffic shows up as a 2-4× CPU spike during peak crawler hours.

Path-preference data is the lever most teams ignore. Each bot has a distinct preference distribution across your URL space, and the preferences map cleanly onto each bot's end goal. GPTBot wants long-form text it can summarize; ClaudeBot wants technical reference material it can answer questions from; PerplexityBot wants commercial pages users compare; Google-Extended wants whatever Googlebot has already touched.

The cadence ranking is its own finding. PerplexityBot has the fastest fetch (minutes from query to crawl, on-demand only). GPTBot revisits high-traffic pages every 2.4 days. Bytespider crawls commercial pages every 1.8 days on retail sites. ClaudeBot sits at 6.8 days, and Google-Extended at 14 days. The implication for content strategy is direct: fresh content gets distributed through GPTBot first, through ClaudeBot inside two weeks, and through Google-Extended on a quarterly cycle. Plan publishing cadence accordingly.

Two of the eleven user-agents we tracked are not subject to robots.txt: ChatGPT-User and Perplexity-User. Both fetch on behalf of a real user query — somebody asking ChatGPT or Perplexity a question, and the assistant fetching your page to answer it. Because the user is the entity making the request, robots.txt does not apply, and any attempt to block these user-agents will break real-user functionality without stopping training data collection (which the assistant already has from the scheduled bots).

The operational read is to stop thinking of AI traffic as "crawler" and start thinking of it as a two-tier system: scheduled bots that you can shape with robots.txt and llms.txt, plus shadow-crawl traffic that you treat exactly like human read traffic — fast caches, sane rate-limits, structured answers in the page source so the assistant has something good to quote.

The findings translate into a small number of concrete moves. None of them are revolutionary — most of them are the same disciplines you would apply for traditional crawl-budget management. The difference is the prioritization: with AI crawlers in the picture, the cost of getting the basics wrong is higher because the surface that depends on the basics is larger and growing.