AI Crawl Economics: Pay-Per-Crawl and Referral Math

AI crawl economics arrived the moment a crawl stopped being a binary permission and started carrying a price. Cloudflare’s Pay Per Crawl beta lets a site owner set a flat per-request fee and answer AI crawlers with HTTP 402 Payment Required — and that single change converts a security question into a revenue question with an arithmetic answer.

The uncomfortable part is that the arithmetic frequently disagrees with instinct. Some crawlers pull thousands of pages for every visitor they send back, which makes free access an obvious subsidy. Others send a visitor for every handful of pages, which makes a fee look like a tax on your own distribution. The gap between those two cases spans nearly three orders of magnitude, so a single site-wide policy is almost certainly wrong somewhere.

This guide is the economics layer only. It covers how Pay Per Crawl actually bills, what the reported crawl-to-referral ratios imply about break-even pricing, what the licensing marketplaces charge per fetch, what blocking costs you in citation visibility, and which numbers in this space are not solid enough to plan against. For the line-by-line robots.txt directives — which named bot goes in which rule — see our AI crawler access control decision matrix, the predecessor to this post. Nothing here restates it.

Cloudflare’s Pay Per Crawl documentation describes an opt-in framework in which a site owner sets a flat per-request price and each AI crawler is assigned one of three actions. The billing handshake runs over a status code the web has had since 1997 and almost never used: HTTP 402 Payment Required. An unpaid crawler requesting a priced resource receives the 402; a crawler willing to pay retries the request carrying payment headers and receives a normal 200 with the content.

Billing integrity rests on identity. Crawlers authenticate through Web Bot Auth, using Ed25519 key pairs and HTTP Message Signatures, so a fee attaches to a cryptographically identified agent rather than a spoofable user-agent string. That distinction matters more than it sounds: the entire prior generation of crawler policy ran on robots.txt directives that are voluntary, unenforced, and trivially ignorable. A signed request is enforceable in a way a robots.txt line never was.

It is also worth naming what Pay Per Crawl is not. General bot-protection products solve a different problem: Vercel’s BotID, for example, is a client-side invisible-CAPTCHA layer aimed at checkout, signup and API abuse, with Basic mode free on all plans and Deep Analysis priced at $1 per 1,000 checkBotId() calls on the Pro plan (custom on Enterprise). That is an abuse-prevention surface, not an AI-crawler allow-charge-block surface. Conflating the two produces policies that block checkout fraud and monetize nothing.

The crawl-to-referral ratio is the only number that makes the allow-versus-charge decision tractable. It asks a single question: how many pages does a platform fetch for every visitor it sends back? A ratio of 5:1 describes a search engine. A ratio of 1,900:1 describes a corpus builder.

Sourcing, stated plainly. The July 2026 figures below come from a secondary compilation that cites Cloudflare Radar, not from the Radar dashboard itself. We could not confirm the individual figures against a live Radar view at the time of writing, so the attribution chain — compilation citing Radar — travels with every number in this section. Read them as a ranking, not as an invoice.

The year-over-year direction is the more interesting story, and it cuts against the prevailing narrative. Anthropic and OpenAI — the two heaviest crawlers on the July 2025 baseline — both improved sharply in twelve months, roughly 20× and roughly 4.4× respectively, which is what you would expect as training-corpus construction matures and retrieval-time fetching — the kind that produces a citation and a click — grows as a share of total crawl volume. If that trend holds, the case for charging weakens over time for exactly the crawlers where it looks strongest today.

Volatility is the caveat that matters most. The same compilation notes that Cloudflare’s own mid-2025 snapshot recorded a materially higher Anthropic figure than the July 2025 number used in its year-over-year comparison, which tells you these ratios move hard with the measurement window. A ratio measured across a training push looks nothing like the same ratio measured across a quiet month. For the underlying traffic composition — including how much of AI crawl volume is training rather than retrieval — see our AI crawler and bot traffic statistics reference, which anchors those figures directly rather than re-deriving them here.

Here is the whole model in one line. If a referred visit is worth V to you and a platform crawls R pages per referral, then each individual crawl earns you V ÷ R in expected referral value. That quotient is also your break-even fee: charge more than it and the fee beats the traffic, charge less and you have priced your own distribution below cost.

The table below normalizes V to $1.00 per referred visit purely so the arithmetic is legible. That figure is our modelling assumption, not data — substitute your own revenue per organic visit and every cell scales linearly. The reported ratios are the secondary-sourced July 2026 figures from Section 02. The comparison column measures each break-even against the $0.01 per fetch general-content midpoint reported across four licensing marketplaces in Section 04.

Two things fall out of that table that no bot-by-bot policy list surfaces. First, the break-even fees at the top of the table are fractions of a cent. The economics of charging heavy training crawlers do not hinge on setting a high price; they hinge on setting any price at all, because almost any price clears a break-even of five hundredths of a cent. Second, the bottom half of the table is not close either. At a 4.7:1 ratio, a fee would need to sit above twenty cents a fetch before it matched the traffic it displaced — above the ceiling of every reported marketplace band in Section 04, and more than four times the ceiling of two of them.

The practical consequence is that a single site-wide fee is almost always wrong. The correct posture is a split: charge where the ratio is in the hundreds or thousands, allow where it is in single or low double digits, and re-measure quarterly, because the year-over-year movement in Section 02 says these assignments have a short shelf life. If your revenue per organic visit is well above $1.00 — common in B2B services and considered purchases — every break-even in the table rises proportionally and more crawlers move into the allow column.

Break-even is only half the calculation; the other half is what the market will actually pay. The figures below come from a secondary research compilation of marketplace pricing dated April 30, 2026. They are reported ranges assembled across four platforms, not published list prices from any single vendor, and they are quarterly data — old enough by August to be a reference point rather than a quote. The final column is ours: a straight reciprocal showing how many fetches a dollar buys at each end of each band.

The same compilation reports implied per-citation values rather than per-fetch ones: roughly $0.05 to $0.50 for general content and $1 to $5 for premium news, a two-to-tenfold multiplier, with encyclopedic and commodity content pricing below baseline because substitutes are abundant. Translated into the per-fetch unit used above, a general-content citation is worth somewhere between 5 and 50 fetches at the $0.01 midpoint; a premium news citation, between 100 and 500. That conversion is ours, and it is the number worth carrying into a licensing conversation — it says plainly that one citation is worth far more than one crawl.

Every fee schedule shows revenue. None of them show the citations you did not earn. A crawler you block cannot quote you, cannot list you among its sources, and cannot send the reader who trusted that citation. That cost is real, deferred, and invisible — which is a reliable recipe for underpricing it.

The scale of what is currently at stake is modest and growing fast. The same compilation cited in Section 02 puts ChatGPT at 0.913% of crawler-driven referral traffic in July 2026 against Google’s 88.08%, having grown from 0.173% a year earlier — more than a fivefold relative increase off a very small base. Both readings are true at once, and which one you weight determines your policy. If you price on today’s share, blocking looks nearly free. If you price on the growth rate, blocking looks like opting out of the channel three years before it matters.

There is also a failure mode that looks like blocking but is not a decision at all: being unreachable by accident. Redirect chains, authentication walls, and archive pagination that only resolves through JavaScript all produce the same outcome as a deliberate block, without the revenue. Before you price access, confirm you are actually serving it — our AI crawler discoverability audit walks the specific failure that hid an entire archive behind 302s. And if the goal is to be cited rather than merely fetched, the factors that actually earn citations are a separate discipline — we covered those in our AI search citation ranking factors study.

The break-even math changes shape depending on how you convert attention into money. What follows is a posture by revenue model, not by bot — the per-bot directives belong in the access control matrix. Run your own revenue-per-visit through the Section 03 formula before adopting any of these as written.

The pattern across those four is worth stating directly: the case for charging strengthens as revenue per visit falls and page count rises, and it collapses everywhere else. That is why publisher trade coverage and B2B SEO advice have been giving opposite recommendations without either side being wrong — they are solving the same equation with inputs that differ by two orders of magnitude. If you want help putting real revenue-per-visit numbers behind this rather than a normalized $1.00, that is the kind of modelling our agentic SEO engagements start with, alongside the analytics instrumentation needed to measure it.

No discussion of crawl economics survives contact with reality without addressing llms.txt, because it is the one lever in this space that costs nothing to pull. It is also the one most often described as if it were a monetization or access-control mechanism. It is neither — it is a plain-text map of your content for machine readers, with no enforcement and no billing.

Two published trackers measure adoption, and they disagree by design. Presenting either alone is misleading, so here are both with their methodologies intact:

• 51.8% (113 of 218 reachable hosts). Measured on August 3, 2026 across a fixed 219-host panel spanning eight sectors, heavily weighted toward developer tools, SaaS and AI companies. Median file size 14.0 KB; 90 of the 113 valid files include the specification’s recommended blockquote summary.
• 8.7% (87 of 1,000). Measured as of June 2026 across the Tranco top 1,000 domains on a monthly-updated tracker, testing for both /llms.txt and /llms-full.txt. Among only the 549 reachable root domains the rate rises to 15.8%, and 15 sites publish the full variant as well. Named adopters in that ranking include Cloudflare, Azure and GitHub.

Those are not contradictory readings. They are a curated developer-skewed panel and a broad traffic-ranked sample, and the gap between them is the finding: adoption is concentrated almost entirely in the sectors whose audiences use AI coding assistants daily. Averaging them would destroy the only useful signal in the data.

The media row is the one to sit with. Zero of fourteen media hosts on that panel publish the file — the sector with the most acute crawl-economics problem is the sector least interested in the voluntary machine-readable summary. That is not laziness; it is consistent. A publisher whose position is that crawling should be paid for has no reason to publish a free index of what to crawl. Adoption tracks whether you want to be read by machines, and media increasingly does not want to be read for free.

Adoption is also not evidence of use. No major provider has confirmed reading llms.txt at inference time, and Google’s own documentation is explicit that the file has no effect on Search — a point we covered in detail when Google confirmed llms.txt does nothing for rankings. Treat it as a cheap bet on agent-side discovery, not as a ranking or revenue mechanism. For the file format itself see our markdown-first content architecture guide, and for the evidence question in full, our companion piece on llms.txt in practice, publishing this week, takes it apart properly.

Every number in this post is somebody else’s measurement of somebody else’s traffic. Your own ratio is the only one that determines your break-even, and it is measurable in a week. The inputs are simple: crawl volume per platform from your server logs, referral sessions attributable to each AI surface, and revenue per organic visit from your existing analytics. Divide, compare against the fee bands in Section 04, and you have a defensible policy rather than an inherited one.

The measurement itself is the part most teams skip, usually because JavaScript analytics cannot see most of it — a crawler that never executes your tag is invisible to the tool you would naturally reach for. Server-side log analysis is the layer that catches it; our companion piece on finding the AI agents GA4 cannot see covers the detection recipe.

Our read on where this goes: the ratios in Section 02 are improving for the largest crawlers by several-fold a year on the reported figures, and the retrieval share of AI traffic is rising while pure training crawls become a smaller fraction of the total. If both trends hold through 2027, the population of crawlers where charging beats allowing shrinks rather than grows — which would make today’s pay-per-crawl land grab a transitional market rather than a permanent revenue line. The publishers who benefit most are those who charge now, at the peak of the imbalance, while keeping the search and retrieval doors open. The ones who get hurt are those who block broadly, lock in a zero-citation position, and discover in two years that the channel they opted out of became the one that mattered.