FrontierMath v2: When AI Benchmarks Get Error-Corrected

FrontierMath v2 landed on June 12, 2026, and the headline most outlets ran with was the leaderboard: Claude Fable 5 reaching roughly 88% on the benchmark’s hardest tier. The more important story is buried in the release note. An Epoch AI audit found small but critical errors in 42% of the original FrontierMath problems — meaning every “state of the art” claim made on the prior version was scored against a test that was wrong about two in five of its own questions.

That matters far beyond the math-benchmark community. Engineering teams pick models, justify budgets, and ship architectures partly on benchmark numbers. If the benchmark itself carried a 42% error rate, the procurement signal it sent was noisier than anyone treating those scores as ground truth assumed. The correction did not change who was winning — it changed how much confidence the numbers deserved.

This guide walks through what v2 actually changed, why a flagship benchmark had errors in nearly half its problems, what the post-correction score jump reveals about the nature of those errors, and a five-question checklist your team can apply to any vendor benchmark claim — not just FrontierMath.

FrontierMath was established in November 2024 by Epoch AI, a non-profit research organization that tracks and forecasts AI progress. It was built with more than 60 mathematicians, including contributors holding 14 International Mathematical Olympiad gold medals and a Fields Medalist among the advisors. Every problem is designed to be “guessproof” — large numerical answers or complex mathematical objects that leave less than a 1% chance of a correct guess without genuine work. It was, by construction, one of the most rigorous evaluations in the field.

That rigor is exactly why the v2 correction is striking. Following an audit, Epoch AI corrected 123 problems in Tiers 1-3 and 12 in Tier 4, and removed a further 5 from Tiers 1-3 and 7 from Tier 4. The net result: the full dataset dropped from 350 problems (300 in Tiers 1-3 plus 50 in Tier 4) to 338 — a base set of 295 plus an expansion set of 43 Tier 4 problems. For a primer on how evaluations like this are constructed and scored, our guide to standard AI evaluation metrics sets the baseline.

The instinct on reading “42% of problems had errors” is to assume something went badly wrong. The more accurate reading is that verifying expert mathematics at scale is genuinely hard, and that AI itself made the audit possible. Early human quality reviews had flagged something closer to 1 in 20 problems — roughly 5% — needing corrections. It was a later AI-assisted audit that surfaced the much higher rate of small but critical errors across the set.

These are not the kind of errors a casual reader would spot. The problems were authored to be brutally hard. Terence Tao, the Fields Medalist, described the difficulty of solving them at all in stark terms, and Timothy Gowers, another Fields Medalist, said the problems he reviewed looked to be at a different level of difficulty from Olympiad problems. When a problem sits that far past the edge of human expertise, a subtle slip in the stated answer or a boundary condition can survive review for a long time.

That difficulty is the point — and it is also why error rates this high are not unique to FrontierMath. Epoch AI itself draws the comparison to ImageNet, the benchmark that arguably launched the modern deep-learning era and which carried its own well-documented label errors for years. The lesson is not that FrontierMath was uniquely flawed; it is that even the most carefully constructed evaluations carry measurable error, and that error sets a floor on how precisely any single score can be trusted.

Epoch AI’s own framing of the v2 results is the most analytically useful line in the release: model rankings on v2 are similar, but scores are higher across the board. Read carefully, that sentence tells you what kind of errors the audit found. If the corrections had simply removed easy problems or added harder ones, scores would have fallen. Instead they rose — which points to errors that were marking correct answers as wrong.

The clearest illustration is the same model measured before and after. GPT-5.5 in its high-effort mode scored roughly 35% on Tier 4 against the uncorrected set; on v2 the same model scores about 72.5%. A near-doubling on an unchanged model is not a capability leap — it is the benchmark no longer penalizing the model for answers it had right all along. The trajectory table below recomputes that delta alongside the other models with a before-and-after figure.

Two things are worth holding lightly here. The v1 and v2 figures come from different reporting moments and, in the secondary cases, carry wide error margins — so the deltas are directional, not surgical. And the AI co-mathematician’s earlier 48% Tier 4 result was achieved without the computational caps applied to the other models, which makes it not strictly comparable. But the shape of the pattern holds: where a model has both a before and an after, the after is markedly higher, and the ordering survives.

That is the genuinely reassuring finding. A benchmark can carry a high error rate and still be useful for ranking models, provided the errors hit all models proportionally. What it cannot do is support confident claims about absolute capability — “the model solves 35% of the hardest math problems” was simply the wrong number, and no amount of hedging would have rescued it.

Epoch AI’s newsletter reports that Claude Fable 5 holds the top spot on v2, at approximately 87% on Tiers 1-3 and 88% on Tier 4. A secondary leaderboard, LM Council, fills in the field below it. Read those two sources together and a careful picture emerges — one that is more interesting than a single “winner.” On Tier 4, Fable 5 leads clearly. On Tiers 1-3, the top is effectively a tie.

The trajectory matters as much as the standings. As recently as early 2026, Claude Opus 4.5 scored below 10% on Tier 4; at FrontierMath’s launch in November 2024, all six models evaluated solved fewer than 2% of problems. Today’s top scores sit near 88%. That is more than a fortyfold improvement in well under two years — a pace that is precisely why Epoch AI expects the benchmark to saturate soon. If you are weighing Fable 5 against GPT-5.5 specifically, our Claude Fable 5 vs GPT-5.5 comparison goes beyond a single benchmark to cost and coding fit.

FrontierMath did not arrive uncontested. OpenAI funded the benchmark’s development and had access to most of the dataset — all but a holdout set — under a verbal agreement not to use it for training. That arrangement became a flashpoint in December 2024, when OpenAI reported roughly 25% on the benchmark with its o3 model. Several mathematicians who had contributed problems said they were unaware at the time of OpenAI’s funding or its level of access.

Epoch AI later built in structural safeguards: Tier 4 holds 20 problems back from OpenAI for evaluation, with access to the remaining problems, so that some portion of the test always stays uncontaminated. It is reasonable to read a line from the contested launch through to the v2 audit — public scrutiny of a high-profile benchmark tends to invite exactly the kind of close inspection that surfaces a 42% error rate. Benchmark contamination and provenance are their own discipline; our guide to reading LLM leaderboards covers how training-data leakage distorts scores.

The most uncomfortable implication of v2 is that a 42% error rate is not extraordinary. Independent analyses of AI benchmarks have reported invalid-question rates ranging from a couple of percent on some math sets to roughly 40% on others, and annotation-error rates on certain text-to-SQL benchmarks have been reported above half the sampled questions. The specific figures vary by source and method, but the direction is consistent: the tests we rank models with carry meaningful, often unmeasured error.

For teams, the practical risk is not the benchmark error itself — it is the gap between a lab score and a production result. Real deployments introduce messy inputs, tool failures, latency budgets, and cost ceilings that no static benchmark captures. A model that tops a leaderboard can still underperform on your actual workload, which is why the better evaluation question is cost per successful task, not raw accuracy. Our piece on measuring AI performance beyond benchmarks makes that case, and our study of model accuracy and benchmark reliability shows how far reported numbers can drift from observed behavior.

The durable takeaway from v2 is a habit, not a number. Before a benchmark score becomes a procurement input, run it through five questions. The matrix below applies them to FrontierMath v2 itself — and the same grid works on any vendor leaderboard you are handed.

FrontierMath v2 scores green on most of this grid — which is the point. A benchmark that publishes its error rate, names the correction, and survives the version change with stable rankings is more trustworthy after the audit than before it. The benchmarks to distrust are the ones that never disclose a version, never publish a QA process, and quietly reshuffle when the test is fixed.

The v2 episode does not mean benchmarks are useless — it means they are evidence, not verdicts. Here is how that translates into how a team should actually choose and operate models.

The practical move for most teams is to stop treating leaderboards as answers and start treating them as starting hypotheses. Pull the two or three models a benchmark suggests for your task class, then run a small, representative evaluation on your own prompts — the work your product actually does — and measure not just accuracy but cost, latency, and failure modes. That is the exact shape of the comparative evaluations in our AI transformation engagements, where model choice is grounded in your workload rather than someone else’s headline. If the benchmark you are relying on is a coding one, our guide to AI coding benchmarks applies the same scrutiny to SWE-bench-style claims.