Tokenizer Variance: Why Identical Prices Cost More

LLM tokenizer variance is the quietest line item in AI cost planning: two vendors can charge the identical dollar-per-million price while their tokenizers segment the same input into materially different token counts — which means the same task produces different bills. The token, not the dollar rate, is the real unit of purchase, and no two vendors sell the same one.

The stakes stopped being theoretical this year. Anthropic’s own documentation now discloses that Claude 4.7 and later models produce roughly 30 percent more tokens than earlier Claude models for the same input text — a same-priced model line that got more expensive per task through a tokenizer swap alone. Mistral’s Tekken tokenizer made the same class of move in the opposite direction. Neither event changed a headline price.

This reference assembles what is actually knowable as of August 24, 2026: measured vocabulary sizes for the tokenizers that publish them, vendor-stated multipliers for the ones that changed, an explicit “not published” row where a vendor discloses nothing, and a three-tier map of which token counts you can verify before a billed call. Every figure is labeled measured, vendor-stated, or independent-and-unconfirmed — because the entire value of a reference like this is knowing which numbers were run and which were merely read.

Every LLM invoice is denominated in tokens, and every vendor manufactures its own. A tokenizer is the compression scheme that turns text into the integer IDs a model actually consumes: a large, well-fitted vocabulary packs more characters into each token; a smaller or differently-trained one needs more tokens for the same string. Two vendors quoting the same dollar-per-million rate are therefore quoting prices in different currencies with no published exchange rate.

Cost trackers — including our own LLM API pricing index — compare headline $/M-token rates because those are what vendors publish. That comparison is necessary but incomplete: it prices the unit without asking how many units your text becomes. A model that looks 20 percent cheaper on the rate card can cost more per task if its tokenizer needs 30 percent more tokens for your content mix.

One method rule governs everything below, because it is what makes a tokenizer reference citable at all: every count must name the exact tokenizer or encoding and the exact text it was run on, and measured numbers must be separated from published ones. This post contains no invented side-by-side sample counts. Where a vendor publishes a vocabulary size in its own config file or library, we report it as measured and name the file. Where a vendor asserts a multiplier in its own docs, we report it as vendor-stated. Where a vendor publishes nothing, the row says not published — never an estimate from a characters-per-token rule of thumb, and never an inference from another vendor’s tokenizer.

The clearest documented instance of tokenizer-driven cost variance sits in Anthropic’s own token-counting documentation: “Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. The same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content and workload shape.” That is a vendor stating, in official docs, that a same-priced model line became measurably more expensive per task purely through a tokenizer swap — at unchanged $/M-token pricing.

The same page instructs developers not to reuse token counts measured against earlier models, and to recount every prompt against the specific model they intend to bill — because the counting endpoint returns counts under the tokenizer of whichever model is passed. Most coverage treated this as a Claude-specific migration footnote. It is better read as the cleanest public evidence of a structural pattern: the billing unit itself can be redefined underneath a stable price.

Independent analyses put the penalty higher for structured content. Trade write-ups reported increases as high as roughly 35 percent on code and JSON payloads specifically — against Anthropic’s blanket ~30 percent figure for general text. Notably, the independent write-ups do not agree with each other: one headline analysis framed the bill impact at 12–27 percent while another put code-heavy workloads at up to 35 percent. That disagreement is itself the finding — there is no single clean multiplier, because the increase depends on content shape. Keep the vendor’s ~30 percent and the independent 35 percent clearly separated; they are different numbers from different sources.

What about Claude’s actual vocabulary? Anthropic has never publicly released its tokenizer — it is exposed only through the counting API, not as a downloadable vocabulary or merge file. The most careful independent reverse-engineering, published July 29, 2026 by researcher Sander Land, estimates roughly 49,000–55,000 tokens (most likely 49,152) for the v3–v4.6 generation and roughly 16,000–20,000 (most likely 16,384) for v4.7 and later. Those figures are explicitly speculative — the author says so himself — and we report them only as unconfirmed third-party estimates, never as Anthropic’s real numbers.

One analytical connection is worth drawing, clearly labeled as our framing rather than a sourced number: a smaller vocabulary is structurally consistent with producing more tokens for the same text, because each merge step compresses less. Anthropic’s vendor-stated ~30-percent-more-tokens change and the independent, unconfirmed estimate of a roughly three-times-smaller vocabulary point in the same direction. That coherence makes the estimate more interesting — it does not make it confirmed.

Anthropic is not an outlier — it is one entry in a ledger. Mistral’s own launch post for Mistral NeMo states that its Tekken tokenizer is approximately 30 percent more efficient at compressing source code than the SentencePiece tokenizers it replaced, with similar gains claimed for Chinese, Italian, French, German, Spanish, and Russian text, and 2x and 3x efficiency gains for Korean and Arabic respectively. Same magnitude as Anthropic’s change, opposite direction, different vendor. Tokenizer swaps routinely move real cost by double-digit percentages while the rate card holds still.

The ledger below collects every multiplier a vendor or named analyst has published for these events. No row is a number this post computed from a fresh tokenizer run; the derived column applies one stated formula, shown beneath the table, and only where a source states a direct token-count change.

*Bill multiplier = 1 + the stated token-count increase, at an unchanged per-token price. It is computed only for rows where a source states a direct token-count change (1 + 0.30 = 1.30; 1 + 0.35 = 1.35). The 12–27% figure in the second row is stated by its source as a bill impact rather than a token-count change, so it is not converted here. Mistral’s efficiency claims are left underived because the launch post does not publish the token counts or sample corpus behind them — converting “efficiency” into a bill multiplier would require assuming a formula the vendor never stated.

OpenAI’s own generational change belongs in the fewer-tokens column too, though it published no blanket multiplier: the move from cl100k_base to o200k_base roughly doubled the vocabulary, and one independent blogger’s own multilingual tests reported dramatic compression gains for non-Latin scripts — his Gujarati sample reportedly dropped from 145 tokens under the old encoding to 33 under the new one. Those are the author’s own measurements of his own sample strings, cited here as an external finding — not counts this post reproduced.

This is the table we could not find anywhere else: all three closed frontier vendors and five open-weight families side by side, each vocabulary figure traced to the actual config file, library source, or paper that publishes it — with an explicit transparency axis. “Measured” means the number was read directly from the vendor’s own published artifact, and that artifact is named in the row. Where a vendor publishes nothing, the cell says so.

Three footnotes the table cannot carry. First, OpenAI’s gpt-oss models use a third, distinct encoding — o200k_harmony — per the model-to-encoding mapping in tiktoken’s own source; its vocabulary size is not separately confirmed here, and it should not be assumed to equal o200k_base’s count. Second, Meta moved from SentencePiece to tiktoken-style byte-level BPE starting with Llama 3, but we publish no Llama 3 vocabulary figure — it could not be verified against an accessible primary source during this research pass, so only Llama 4’s 202,048 appears above. Third, the Gemini row’s 262,144 figure is an inference from the openly released Gemma weights that Google has never made about the Gemini API — it stays labeled unconfirmed.

Even among published vocabularies the spread is roughly twofold — about 100 thousand to about 202 thousand entries. Vocabulary size alone does not determine token counts for your content mix (the training corpus and merge strategy matter as much), which is exactly why the method rule exists: the only trustworthy comparison for your workload is running your own corpus through each tokenizer you can actually obtain, and using each vendor’s own counting surface where you cannot.

The matrix’s most consequential column is the last one: whether you can know your token count before a billed call. The industry splits into three tiers, and the tier a vendor sits in determines how much of your cost model rests on trust.

The Tier 3 surfaces are still worth using well. Anthropic’s count_tokens endpoint accepts the identical payload shape as the Messages API — system prompt, messages, tools, images, PDFs, thinking blocks — and is free to call, with its own rate limits separate from message creation: 2,000 requests per minute on the Start tier, 4,000 on Build, 8,000 on Scale. Two caveats from the same official page: the count may include tokens Anthropic adds for system optimizations that are not billed, and the endpoint does not apply prompt-caching logic even when cache_control blocks are present — so a pre-flight count cannot estimate a cache-discounted bill.

Google documents a countTokens API that returns a breakdown including input, output, thought, cached, and tool-use token totals, and its Gemini API token docs offer the informal rule of thumb that a token is equivalent to about 4 characters of English text. The media rates published alongside that countTokens reference are concrete where text is not: images at or under 384px cost 258 tokens, larger images 258 tokens per 768×768 tile, video 263 tokens per second, audio 32 tokens per second. Whether the countTokens call itself is billed is a detail we could not confirm from a primary source for this reference, so we make no claim either way — check the current pricing page before wiring it into a high-frequency preflight loop.

The forward projection writes itself: as agentic workloads push token volumes up and routing layers arbitrage between vendors per request, the verifiability gap becomes an operational risk, not a curiosity. A router that compares vendors on $/M-token without normalizing for tokenizer variance is optimizing the wrong number — and for two of the three closed vendors, the normalization factor cannot be computed offline at all.

Tokenizer variance is steepest across languages. A June 2026 academic study of multilingual tokenization put the structural problem plainly: state-of-the-art multilingual LLMs often use byte-level BPE tokenizers that structurally favor high-resource languages and Latin scripts — inflating inference costs for speakers of underrepresented languages as a property of tokenizer design, not of any one vendor’s pricing.

Vendors are narrowing the gap generation over generation — that is what Mistral’s Korean and Arabic claims describe, and what the independent o200k_base analysis observed for Indic scripts. But the popular inversion of this fact — “prompt in a cheaper-to-tokenize language and save money” — does not survive testing. An April 2026 paper running SWE-bench Lite tasks found the token-efficiency advantage of Chinese prompting is not observed uniformly: MiniMax-2.7 needed 1.28x more tokens for Chinese prompts while GLM-5 needed fewer — the same trick flipping sign between two models in the same study. The paper also found success rates for Chinese prompting generally lower than English across all models tested, meaning a genuine per-token saving can be erased by retries.

The interpretation matters more than any single figure: language arbitrage is tokenizer-specific, direction included. Whether a given language is cheap or expensive is an empirical property of one vendor’s vocabulary, measurable only against that vendor’s tokenizer — one more reason the “name the tokenizer, name the sample” discipline is not pedantry but the difference between a citable number and a folk claim.

Everything above reduces to four operating rules for anyone running LLM spend at meaningful volume.

Rule 4 deserves its own reference: the derived-numbers QA pattern covers gating tables that contradict their own formulas — the failure class a misattributed token count belongs to. On the spend side, tokenizer variance is one input among several; our AI inference FinOps playbook covers the full stack, and anyone running an agentic content operation can see token cost per shipped unit tracked honestly in our agentic pipeline yield ledger. For teams that want the comparative eval run against their own corpus and workload shape rather than a rate card, that measurement discipline is where our AI transformation engagements start.