AI-Written Web Text Helps Small Models, Then Starts to Hurt

A paper submitted to arXiv on September 30, 2026 asks a question every lab now faces: what happens when the web you crawl for training data is partly written by the models you trained last year? The authors, from the University of Maryland and the AI detection company Pangram Labs, label almost a third of quality-filtered August 2026 web tokens as AI-generated, then pretrain 800 small models on mixtures of human and AI text. Their finding is a sign change. AI text helps a model that lacks human data, stops helping at the compute-optimal budget, and raises loss beyond it.

Editorial note: Prepared October 3 as an October 1, 2026 dispatch from the paper’s arXiv abstract and HTML text, version 1. One paper, not yet peer reviewed. It makes no claim about how search engines treat AI text, and neither does this post.

The paper, How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text, starts by measuring. The authors take Common Crawl snapshots, apply the FineWeb quality filter that many open pretraining pipelines use, and run two detectors over what survives: a small open classifier to label everything, then Pangram’s own detector as a second check on the confident cases. The result is the chart below. The two later bars are the authors’ forecasts, not measurements.

Two things about the measurement matter for everything after it. The share is of tokens that pass a quality filter, and the paper reports that AI text survives filtering more often than human text does, so the filtered share is higher than the raw share. And the label comes from a detector sold by one of the authors’ employers. The paper is open about both; a reader should carry them forward.

The experiment holds a human corpus fixed and adds AI text to it, up to 64 AI tokens for every human token, across models from 19.9 million to 973 million parameters. Loss is measured on held-out human text from three sources and, separately, on held-out AI text. The design is what distinguishes the paper from earlier model-collapse work: the AI text is not generated by the model under test or curated to help, it is whatever the web contained, from many models, written for people.

The pattern the authors describe is consistent across sizes. For a model starved of human data, adding AI text lowers loss on human text at first, then the benefit saturates and reverses as more is added. For a model already trained at or past the compute-optimal budget of about 20 human tokens per parameter, AI text raises loss almost immediately, and does so more sharply at larger sizes, while the same number of fresh human tokens keeps lowering it. The paper notes that today’s production models train far past that budget, citing one open model at 1,100 tokens per parameter, which is the regime where the harm applies.

Existing scaling laws cannot express this, and the paper says why. The Chinchilla law treats an AI token as a human token. Laws built for repeated data let a token’s value fall towards zero but never below it. The authors fit a law with separate benefit and harm terms, so the value of an AI token can change sign, and which reduces to Chinchilla when no AI text is present. Fit on the smaller models, it predicts the effect of AI text on models up to 3.6 times larger with 41% lower error than the best existing law, on the authors’ own evaluation.

The law yields the paper’s most quotable figure. Training on unfiltered web text at August 2026’s AI share requires 1.6 times the compute of training on the human subset alone, at the compute-optimal budget, and the gap widens with more human data. At the forecast shares of 42% and 51%, the multipliers become 2.1 and 3.0. Those last two numbers rest on a forecast and a fitted law, and should be quoted as such.

The abstract ends with four recommendations for anyone building a pretraining corpus. They are short enough to table.

The third row is the one with consequences outside research labs. If a mixed validation set hides the harm, then a team that fine-tunes on scraped web text and checks only an aggregate loss will not see the problem it has introduced. Our decision guide on synthetic training data makes the same point from the curated side: generated data is a tool with a target, not a free supply.

It measures loss, not capability. Every result is a held-out cross-entropy on text; the paper does not report whether the models answer questions worse or follow instructions worse. Lower loss on human text is the standard proxy, and the authors use it as one, but a reader should not translate the 1.6 figure into a benchmark score.

Its largest model is under one billion parameters. The scaling law is validated on models 3.6 times larger than those it was fit on, which is still far below production scale. Whether the harm term keeps its shape at tens of billions of parameters is a prediction the paper makes, not a measurement it reports.

Its labels come from a detector. A token is “AI-generated” when Pangram says so, and the company that sells Pangram co-authored the paper. The authors disclose this and release the corpus with its labels so others can relabel it. Until someone does, the share figures are one detector’s view.

And it says nothing about search. The paper is about what AI text does to a model that trains on it. It does not test, and does not claim, that search engines rank AI-written pages differently; the question of what publishers should do with AI text is covered in our note on Google’s updated guidance on reviewing AI content, which is a separate matter with a separate primary. For the data-supply side, our piece on publishers and Common Crawl is the context for who decides what gets crawled.