#model-evaluation

A free, anonymous 1M-context reasoning model appeared on OpenRouter on August 20. Nobody has claimed it, and its no-training promise is a one-off exception.

We collected 42 vendor-published benchmark rows from 9 vendors and checked what each page discloses: version, subset, harness, effort. Full table in the post.

Four levers decide what a vendor benchmark table means: which benchmark version, which task subset, what shape the metric takes, and who got a column.

Anthropic says its Fable 5 retune cut biology-related fallbacks about 85% across product surfaces. A rare published guardrail false-positive figure.

Kimi K3 leads open-weights models on intelligence, but its hallucination rate climbed to 51%. Why you must run your own evals before adopting it.

ARC Prize independently administered Claude Opus 5's 30.16% ARC-AGI-3 result. Almost every other launch-day benchmark number is vendor-run on a vendor harness.

Between July 17 and 23, seven notable models shipped from five vendors — most efficiency or positioning plays, not capability jumps. A triage framework.

Four AIs hit a perfect IMO 2026 score in one VC's own test, not official IMO grading. What benchmark saturation means for how you should judge models.

Claude Fable 5 tops SWE-bench Verified at 95%, but 99 of 100 results are self-reported and the scaffold gap can exceed 28 points. How to read the numbers.

Epoch AI found errors in 42% of FrontierMath problems and shipped v2 on June 12, 2026. Scores jumped, rankings held — here is what that means for model choice.

Opus 4.8 tops the Artificial Analysis index, but GPT-5.5 still leads Terminal-Bench. An evidence-graded roundup of the first 48 hours of independent evals.

How to read AI model leaderboards without being fooled by benchmark contamination, eval gaming, and cherry-picked MMLU, GPQA, and SWE-bench scores.