What a 14 Times Faster AI Claim Actually Measures

A speedup claim is a ratio, and reading one means finding the number underneath it. Between August 13 and August 27, 2026, three separate AI speed claims landed — two of them carrying the identical headline figure of fourteen times faster — and they measured three different things. One disclosed every condition it was taken under. One disclosed no baseline at all. The third describes the same base model as the first and is still not comparable to it.

This is deliberately a narrower subject than it might look, so it is worth drawing the line in the first two paragraphs. A great deal has been written lately about what a vendor number measured — which subset, which harness, which version, whether the leaderboard was contaminated. Our own archive holds several of those, and the leaderboard-methodology guide is the one to read if a ranking is what you are trying to trust. A leaderboard tells you where something ranks. A speedup tells you how much faster something is than something else you have to go find — and this post is about the something else. Every post in that genre is about what a number measured; this one is about what a number was divided by. That is a different question with a different failure mode, and the rest of this post is a method for answering it.

What follows is a method rather than a verdict. Nothing here accuses a vendor of dishonesty, and the strongest single exhibit is a primary source that published every condition it measured under and still produced a headline nobody can compare to anything. We work through the pair, then the taxonomy of things a speedup can be measured against, then the conditions that move a ratio without anyone changing the product, then what a real disclosure standard requires, then a four-question checklist for the next claim you see. The worked example is video generation, and it is confined to one section — the compilers, package managers and inference engines carry the rest, because the same arithmetic applies everywhere.

Start with the coincidence, because it does the teaching for free. In a fourteen-day window in August 2026, three parties published speed claims about AI inference. Two of them used the same headline number. None of the three is comparable to either of the others, and the reasons differ in each pairing.

On August 13, 2026, OpenAI previewed an Ultrafast tier for GPT-5.6 Sol and described it as up to fourteen times standard speed. We covered that preview when it landed, and the relevant detail for this post is a negative one: our write-up of the Ultrafast preview records no disclosed baseline workload, no prompt set and no reasoning-effort setting. Even the Standard-tier figure that the multiple would have to be measured against is an inference we drew from two separate vendor numbers, and that post labels it as our inference rather than a vendor statement. We are not upgrading that hedge here.

Then, on August 27, two claims arrived on the same day about the same open-weights base model. FastVideo — the Hao AI Lab team at UC San Diego, working with Nuva Lab and the NVIDIA FastGen team — released FastH3 Preview v1, a four-step distilled derivative of MiniMax’s H3-Base weights, headlined “Up to 14x speedup on a single Nvidia Blackwell GPU”. The same day, the inference platform fal announced H3 Max, another post-trained derivative of the same base model, at roughly thirty-five times the throughput of its own hosted official MiniMax H3 endpoint.

Read down that list and the disclosure spectrum is the finding, not the numbers. Rows one and three share a base model, a publication date and a modality, and are still not comparable, because the clip length, the hardware, the method and the baseline all differ. Rows one and two share a headline number and nothing else — one is a video-generation pipeline timed end to end, the other is a text inference tier. If you saw all three in a feed on August 28 you would have had no way to rank them, and ranking them was never a coherent operation in the first place.

One correction belongs here rather than in a footnote, because it is the most common error in circulation about the first row. FastH3 Preview v1 is FastVideo’s release, not MiniMax’s. The primary’s own citation block gives the author as the FastVideo Team; its acknowledgements thank MiniMax for releasing H3-Base, which is the posture of a team crediting the weights it post-trained rather than an author list. The Hugging Face card for the recommended checkpoint calls it the FastH3 Preview v1 checkpoint from FastVideo. MiniMax published the base weights and the community licence the derivative inherits; it did not publish this speedup.

The reason to spend a paragraph on attribution in a post about arithmetic is that the two errors are the same error. On August 28, 2026, the aggregator Digg published the story under the headline “MiniMax Announces Fast H3 v1 Video Model”, with a dek reading “MiniMax celebrates release of Fast H3 v1 video model with partner labs” and, per the page’s own counter, 253.4K combined views. Its body text is closer to right — it reports that MiniMax posted congratulations to the Hao AI Lab, Nuva Lab and NVIDIA FastGen teams — so this is a headline compressing a congratulation into an announcement, not a fabrication. Note that Digg is a rolling aggregator and the page may since have changed; the version quoted here is the one fetched on August 30, 2026. A headline that compresses “MiniMax congratulated the labs that built this” into “MiniMax announces” is doing exactly what a headline does when it compresses “14.38× at the 15-second shape, on one B200, against our own dense baseline” into “14x faster”. Same operation, same information loss.

The structural point is almost embarrassingly simple. A benchmark score is a measurement: one number, one system, one set of conditions. A speedup is not a measurement at all — it is a division. It requires two measurements, and the second one is a choice the claimant makes. Change the choice and the number changes without anything about the product changing at all. That is why the usual reading advice does not transfer: checking whether a benchmark was run honestly tells you nothing about whether a ratio’s baseline was well chosen, because a perfectly honest measurement divided by a favourable baseline still produces an incomparable multiple.

Our archive already covers the neighbouring questions, and the division of labour is worth stating so you can go to the right place. Reading a vendor benchmark table that shows a loss works through four levers inside a quality table — version, subset, metric shape, and who got a column. This post is about the fifth lever that table cannot show you: the denominator of a ratio. Our reproducibility audit of 42 vendor benchmark rows asked whether a buyer could re-run what a vendor published. And the voice-AI benchmark read happens to hold a multiple that moves with conditions in a completely different modality, which is a useful reminder that this is not a video problem or a compiler problem.

Before going further it is worth being precise about what this article can and cannot claim, because a sloppy absence claim in a post about rigour would be self-refuting. “Nobody publishes the denominator” is false, and it is not a sentence we will write. FastVideo publishes its denominator in the same table as its ratio, under a column labelled as the speedup over Base H3. Astral links its benchmark document from the headline sentence of the uv README. The honest version, scoped to what was actually read, is narrower and more useful.

Across the speedup claims examined for this article — FastVideo’s FastH3 Preview v1, fal’s H3 Max as recorded in our own August 27 coverage, Microsoft’s TypeScript 7 GA and 2025 native-port posts, Astral’s uv README and benchmark document, vLLM’s Blackwell and InferenceMAX post, and NVIDIA’s TensorRT-LLM deployment guide, each read on August 30, 2026 — every headline stated a multiple, and the matched run that produced it lived somewhere below the fold. Some headlines name a class of baseline (fal names its own hosted H3 endpoint; uv names pip; vLLM names Hopper GPUs). None of those eight put the matched conditions in the headline. The failure is in the compression, not in the disclosure. We did not survey the field, and this is what those eight documents show. Nothing in this post asserts a frequency beyond them.

That is a genuinely more interesting finding than a disclosure scandal would have been. It means the information you need mostly exists, and the work is retrieval rather than investigation — which in turn means a reader with four questions and five minutes can usually resolve a claim that looked unresolvable. Section 09 is those four questions.

This is the one section in this post whose subject is video generation. It earns the space because the pair is unusually clean: two derivatives of one open-weights base model, two speedup claims, one calendar day, and a complete disclosure asymmetry between them. Everything after this section is compilers, package managers, inference engines and standards documents, because the arithmetic does not care about the modality.

Side A is FastH3 Preview v1 from FastVideo. Side B is H3 Max from fal, which we covered on the day — our August 27 write-up of H3 Max has the launch detail, and this post takes only three multiples, the shape of the comparison set, and the serving hardware from it. The base model itself is MiniMax H3, which launched on July 31, 2026 with native audio; both sides post-trained it, and neither of them is MiniMax.

The asymmetry in the final column is the load-bearing observation, and it is a disclosure difference rather than a suggestion that either number is wrong. Side A publishes its warmup, its exclusions, its statistic and its sample count. Side B publishes none of those four, and publishes no absolute seconds for either half of its ratio. fal also published two further multiples in the same announcement — roughly fifteen times faster than anything with comparable quality, where comparability is set by an internal study that is not published, and a figure of more than fifty times that the announcement attributes to Design Arena. Three multiples, three different denominators, one press release.

Now the part that makes this section worth reading even if you never touch a video model. FastVideo published twelve speedup cells, not one, and the twelve cells disagree with each other in a way that demolishes the headline’s implied generality — honestly, legibly, and in the same table.

Read across rather than down. The headline number, 14.38×, is the largest of the six cells. Change nothing but the clip length and it becomes 8.16× — a 1.76× swing from clip length alone. Across all six VSA cells the smallest is 6.65×, so the headline is 2.16 times the floor of its own table, and every one of those six numbers is equally true. Change nothing but the GPU count, from one B200 to four at the same fifteen-second shape, and it falls to 12.48×, because parallelism helps the baseline too and the ratio shrinks as you scale. Same weights, same kernel, same team, same measurement protocol, same day.

Nobody misled anyone. The table is right there, under the headline, and it is more complete than most vendors publish. That is precisely why it is the strongest exhibit available: it demonstrates that the compression from a six-dimensional result to a single multiple destroys comparability even when the publisher is scrupulous. A speedup claim does not need a bad actor to become unusable. It only needs a headline.

FastVideo also publishes the ablation that separates the two mechanisms behind the speed, and it is the honest analytical read of the length sensitivity above. The dense variant of the same four-step student sits at 7.24×, 7.52× and 7.43× at the five-, ten- and fifteen-second shapes on one B200 — essentially flat across clip length. The sparse variant climbs from 8.16× to 14.38× across the same three shapes. So step distillation, which cuts the transformer calls from 49 to 4, contributes a roughly constant factor, while the video sparse attention at 90% sparsity contributes a factor that grows with sequence length. At the fifteen-second shape, sparse attention is worth 1.94 times what dense is (14.38 ÷ 7.43). That is a mechanism, not a measurement trick — and it means the headline number is length-sensitive by physics, which is the opposite of an accident.

One last derived observation from the same primary, because it shows how far a label can drift from what it names. FastVideo states that its five-, ten- and fifteen-second shapes contain 124, 243 and 345 frames, and that the pipeline runs at 24 FPS. Divide and the shapes are 5.167 s, 10.125 s and 14.375 s of actual footage — so the shape called “15 s” is fourteen and three-eighths seconds long. Put those footage durations next to the published eight-GPU end-to-end times and the sub-realtime claim resolves cleanly: it is true, and it is true for exactly one of the three shapes.

Once you start asking what the denominator is, the answers sort into a small number of kinds — and the kinds matter more than the magnitudes, because two multiples of different kinds are not commensurable no matter how carefully each was measured. Four of the six examples below are not video, which is the point.

The taxonomy has an immediate use: it tells you which claims can be put in a league table and which cannot. A type-1 multiple is a statement about an engineering team’s own work. A type-5 multiple is a statement about a chip. Averaging them, ranking them, or citing one to rebut the other is a category error, and it happens constantly in feeds and in slide decks, because the multiple looks like a unit and is not one.

Note also what the taxonomy does to the August 13 claim from section 01. With no baseline workload, prompt set or effort setting published, OpenAI’s “up to 14×” cannot be placed in any of the six rows. That is not a rhetorical trick; it is the practical consequence of the omission. An unplaceable multiple can be believed or disbelieved but it cannot be compared, which means it carries no information a buyer can act on beyond “faster than before, by some amount, under some conditions”.

Knowing the denominator type is not sufficient, because within a single type the number still moves. Every row below is measured from a single primary source with one condition changed, wherever a primary published enough to make that possible. Read the swing column as the answer to a specific question: if a vendor moved only this one dial, how much would the headline change?

The first row deserves more than a cell, because it is the cleanest demonstration in the whole set that a speedup needs to say whose speed it is. NVIDIA’s own TensorRT-LLM deployment guide publishes a concurrency sweep on identical hardware: sixteen GPUs, one model, nothing changed but the number of simultaneous requests. Aggregate output throughput moves from 201.2 to 3,823.9 tokens per second — a factor of 19.0 — while per-user throughput falls from 55.2 to 16.1 tokens per second and median time-to-first-token rises from 1,769 to 4,587 milliseconds, a factor of 2.6.

A vendor could truthfully publish “19x throughput” from that table, and a customer sitting in front of the resulting deployment would experience the machine as roughly three times slower than the low-concurrency configuration. Nothing about the model or the silicon changed. This is why the percentile question is not pedantry, and why our own latency dataset publishes P50 and P95 rather than a single figure — a median hides the tail that users actually feel, and an aggregate hides the individual entirely.

One more shape worth recognising, because it turns up whenever a speed gain is expressed as a saving rather than a multiple. Some vendors publish percentages instead of ratios — a refactoring benchmark reported as “83% faster”, for instance, which we worked through in the JetBrains refactoring-skill read. A percentage carries exactly the same denominator problem in a shape that makes it harder to spot, because a percentage looks like a property of one thing rather than a relationship between two. Convert it to a multiple before you compare it to anything: 83% faster is about 5.9 times, and stating it that way immediately prompts the right question — faster than what?

If the video pair feels like a special case, here is the same structure in a domain almost every reader has on their laptop. TypeScript’s native compiler rewrite has been benchmarked repeatedly by its own team on the same public codebase — the vscode repository — across the March 2025 native-port announcement and the TypeScript 7.0 GA post, and has been published at five different multiples, every one of them defensible and every one of them measuring something slightly different.

Microsoft’s headline across both announcements is “10x” — a number that appears nowhere in the five bars above, sitting instead between the third and second of them. Its own GA body text is more careful than the headline, describing optimisations that “typically yield speedups between 8x and 12x on full builds”, and the full GA table ranges from 7.7× to 11.9× across five codebases at the default checker count, then from 10.6× to 16.7× across the same five at --checkers 8. The vendor is not overstating; the headline is a rounding of a range, and the range is published two screens down.

Two honest notes on that chart. The 7.2× bar is a figure we found reported as the VS Code team’s own --noEmit measurement by Visual Studio Magazine; we could not locate a first-party page for it, so treat it as secondhand reporting rather than a vendor statement. And the machine matters here more than anywhere else in this post: the difference between 11.9× and 16.7× is purely the number of checker processes, and the GA post says only that both tables were run on the same machine — it never says what machine, which core count, or which operating system. To Microsoft’s credit the post publishes the caveat itself, noting that these codebases “get a better speedup from dedicating more cores, but results will differ across projects and underlying machines”. It publishes the lever and the warning and omits the hardware. That combination is the industry norm at its better end.

If you are actually deciding whether to move a codebase, our TypeScript 7 migration playbook is the post for that. Here the compiler is only a specimen, and the specimen’s value is arithmetic: this is what it looks like when one product, one repository and one honest team produce five different true multiples.

That last row is teaching material, not an error, and it is worth being explicit about why. Divide the published times and date-fns comes out at 9.29×, not the published 9.5×. The explanation is almost certainly benign: 6.5 seconds and 0.7 seconds are one-decimal roundings of the real timings, and an unrounded 0.68 seconds would produce 9.5× exactly. The tRPC row shows the same artefact in the opposite direction — 9.167× recomputes to 9.2× and is published as 9.1×. This is not a fabricated number; it is what happens when a table publishes one-decimal seconds alongside two-significant-figure ratios.

The general rule that falls out of it is the useful part. When a table publishes rounded inputs and rounded ratios, the ratio column cannot be recomputed from the table. Every other ratio we checked does recompute: all twelve FastH3 cells and all ten TypeScript 7 GA cells land within ±0.03 of their published values. And FastVideo solved the whole problem with one sentence stating that its speedups use the unrounded timings. That sentence costs nothing and it is the single cheapest disclosure improvement available to anyone publishing a benchmark table.

Three more figures from the TypeScript GA post, because they show the same author being rigorous and loose in the same document. Slack’s CI type-check going from about 7.5 minutes to 1.25 minutes is a fully specified 6.0×. Canva’s first-error-in-editor time going from roughly 58 seconds to about 4.8 seconds recomputes to 12.08×. But the same post also reports that the new language server reduced failing language-server commands by over 80% and server crashes by over 60% compared with the previous version — two rates with no sample size for either, no time window, and no statement that they come from the same telemetry population. That is not dishonesty. It is the ordinary asymmetry: the numbers a team benchmarks get conditions, and the numbers a team telemeters get percentages.

Four more claims, chosen because each one shows a different relationship between a headline and the disclosure underneath it. Two of them are, on the evidence, better-disciplined than the headline suggests — which is why this post is not a prosecution. The primaries are worth opening yourself: uv’s benchmark document and vLLM’s InferenceMAX post are both short, and both publish more than they had to.

The transferable sentence from that set belongs to vLLM, and it is the most quotable line in this entire subject: single-point metrics can be misleading. A speedup is a single-point metric by construction — it collapses a curve into one number and then discards the axes. The constructive alternative is in the same post: publish the frontier. Give a reader throughput against per-user latency across configurations and the reader can find their own operating point instead of trusting yours.

There is a pattern here that we keep running into from different directions, and it is worth naming as a trend rather than a set of anecdotes. The disclosure in this industry is getting better at the primary-source layer and no better at all in the compression layer. FastVideo publishes twelve cells and a footnote about rounding. Astral publishes four scenarios, a harness, an OS and a workload. vLLM publishes a daily-refreshed benchmark and a rationale for preferring curves to points. None of that changes what happens to the number when it reaches a headline, an aggregator, or a slide — and the aggregator step is where our own reading of a widely-cited study whose denominators were never stated landed on the same failure. The forecast is unexciting and probably correct: primaries will keep improving, headlines will not, and the skill of retrieving the denominator will keep appreciating in value.

It is easy to complain about disclosure and hard to say what adequate disclosure would be. Fortunately a standards body has already written it down. The MLPerf Inference rules and the accompanying MLCommons submission rules specify, in binding detail, what a performance claim must carry before it can use the name. Read them once and every vendor blog post looks different afterwards.

Start with the statistic, because it is the first thing MLPerf takes away from you. You do not get to pick it: the scenario determines it.

Compare that with the protocols in this post. FastVideo reports the median of three timed requests, which is a better-documented protocol than most vendor blogs manage and would still not qualify as an MLPerf datacenter result: three runs, a median rather than a percentile, and nothing approaching a ten-minute minimum duration. That is a difference in purpose rather than a failing — a research blog post is not attempting a standards submission, and MLPerf Inference does not have a video-generation benchmark that matches this workload anyway. But it does calibrate what “we measured it carefully” means at different levels of ambition.

The rules also explain why the percentile is mandated. The early-stopping provision exists, in the document’s own words, with the penalty that “the effective computed percentile will be slightly higher”, because in runs with few queries “there is a higher probability that a particular run will, by chance, report a lower latency than the system should reliably support”. A standards body wrote a rule specifically to stop a small-n run from flattering itself. A median of three is exactly the shape that rule was written about.

Our own reading of the MLPerf Training v6.0 results covers what the published numbers said about the 2026 hardware race; this section is a different document entirely — the Inference division’s rules about what a submitter must disclose, not any result. The reason the rules are the interesting half here is that they encode a philosophy of measurement that a marketing team can borrow without joining a consortium. The submission rules require that on-premise systems be described “in sufficient detail to enable third parties to build a similar system”. That is the test. Not “did we tell the truth” but “could someone else reconstruct the conditions”.

The practical payoff of everything above is short, and it is deliberately four questions rather than a rubric, because a rubric nobody runs is worth less than four questions somebody does. Each one is answerable from a primary source in about a minute, and the evidence in this post says the answers are usually there — below the fold, in the table, in the caption, or in a linked benchmark document.

Two notes on using this in anger. First, the discipline is the same one that applies to any performance number you report internally, which is why we treat it as measurement work rather than as press-release literacy — it is the same habit we bring to analytics and measurement engagements, where a percentage lift with an unstated baseline period causes precisely this failure and costs considerably more than a misread benchmark. Second, if a claim survives all four questions, believe it — for the conditions it names, and not one condition further.

Where this is heading is worth a sentence of forecast, because the trend line is fairly clear. The number of speedup claims in circulation is rising faster than the number of independent re-runs: we could not locate an independent wall-clock verification of either side of the video pair in this review, and that is normal rather than remarkable. As distilled derivatives of open-weights models proliferate, so does the population of claims about the same base model measured under incompatible conditions — which means the comparability problem gets structurally worse even if every individual publisher gets more rigorous. The workable response is not more scepticism. It is the habit of asking, every time, what the number was divided by.