Kimi K3 Benchmarks: The Hallucination Number Nobody Charted

The Kimi K3 benchmarks that made the launch coverage tell one story: a 57 on Artificial Analysis’s Intelligence Index, the leading open-weights model on the board, and 4th overall as of July 27, 2026. The number that didn’t make the charts tells another. On the same organisation’s knowledge and hallucination benchmark, K3’s hallucination rate rose from 39% on Kimi K2.6 to 51% — while its accuracy rose from 33% to 46%.

Read those two movements together and you get an uncomfortable characterisation of the model: K3 is simultaneously more often right and more often confidently wrong. Its predecessor abstained more. K3 answers more, gets more of them correct, and — of the questions it still doesn’t know — guesses rather than declines at a noticeably higher rate. For a chat assistant, that trade may be fine. For an agent writing to a CRM, drafting client-facing copy, or summarising a contract, it is a materially different risk profile than the headline Index score implies.

This guide is not a general survey of hallucination benchmarks — we’ve covered how hallucination benchmarks are measured across models separately. It is about one specific, checkable case: what K3’s own numbers say, what Moonshot chose not to measure, how far the same benchmark moves when someone else runs it, and the eval-before-adopt gate that turns all of that into a decision rather than a vibe.

Artificial Analysis published its K3 assessment on July 17, 2026, the day the hosted API went live and ten days before the open weights shipped on July 27. Its own article states that K3’s AA-Omniscience accuracy rate rose from 33% to 46% against K2.6, that the hallucination rate rose from 39% to 51%, and that the composite AA-Omniscience Index moved from +6 to +18. Those three numbers come from the evaluating organisation’s own write-up, not from a press relay, and independent tech press reported the same movement the day before publication.

The two movements are close in magnitude — accuracy up 13 points, hallucination rate up 12 — but they are not equivalent in consequence. An accuracy gain is realised by the user who asks a question the model happens to know. A hallucination-rate increase is realised by the user who asks a question the model doesn’t know, and who now receives an assertion instead of a hedge. Those are different populations of query, and in production work they are rarely the same people.

None of this makes K3 a bad model. It is the leading open-weights model on Artificial Analysis’s open-source board as of July 27, 2026 — a 57 against 51 for the next-placed GLM-5.2, with MiniMax-M3 and DeepSeek V4 Pro tied at 44 behind that. It sits 4th on the overall Intelligence Index, behind Claude Opus 5 at 61, Claude Fable 5 at 60, and GPT-5.6 Sol at 59. (Artificial Analysis’s own July 17 article ranked it 3rd; Opus 5 launched in the interim and moved the rank without moving K3’s score. Date-stamp every one of these figures — the index is live and it moves.)

What it makes K3 is a model whose adoption decision cannot be read off a rank. That is the entire argument of this piece, and the reason we build a standing eval harness to qualify new models before any of them touch client work.

AA-Omniscience is built on 6,000 questions spanning 42 topics across six domains — Business, Health, Law, Software Engineering, Humanities & Social Sciences, and Science, Engineering & Mathematics — derived, per the methodology page, from authoritative academic and industry sources. The question set and an accompanying paper are published openly, which is what makes the numbers checkable rather than merely quotable.

The design decision that matters here is that it reports three separate metrics rather than collapsing to one. Most coverage carried one of the three.

For context on how hard the metric is, the benchmark’s original write-up in November 2025 found the leading model of the day topping the Index at just +4.8, one of only three models to score above zero at all. Most frontier models, in other words, produced more confident wrong answers than correct ones on net when refusal was available to them. K3’s +18 is a real improvement against that backdrop. It is also, as the next section shows, an improvement that partly conceals the thing it’s measuring.

K3’s Omniscience Index tripled, from +6 to +18. That is the number a procurement deck would carry. It rose because the Index weights correct-answer volume against hallucinated wrong answers, and a 13-point accuracy gain outweighs a 12-point hallucination-rate increase in that arithmetic. The composite is doing exactly what it was designed to do. The problem is what a reader infers from it.

There is a further derivation worth doing yourself, because it changes the tone of the finding. Artificial Analysis defines the hallucination rate as incorrect answers divided by the sum of incorrect, partial and not-attempted answers. If those four outcome categories — correct, incorrect, partial, not-attempted — are exhaustive, then the denominator is simply everything that wasn’t correct, and you can recover the share of the whole question set that drew an outright wrong answer.

That reframing is more useful than the headline. K3 did not dramatically start fabricating more; it dramatically stopped declining. For anyone building agentic workflows, the operational translation is direct: the guardrail you need is not a smarter model, it is an explicit abstention path — a way for the model to return “insufficient information” that your orchestration layer treats as a valid, routable outcome rather than a failure.

One mechanical confound is worth naming and then not overclaiming. Artificial Analysis also reports that K3 consumed 21% fewer output tokens than K2.6 across the same Intelligence Index run — 132 million against 166 million. Terser models have less room for hedging language and caveats, and it is plausible that some of the hallucination-rate movement is a byproduct of that concision rather than a change in epistemic behaviour. That is our hypothesis about a correlation, not a finding either organisation has published, and it is exactly the kind of thing your own harness can test in an afternoon and a leaderboard cannot.

The Kimi K3 GitHub repository is not shy about evaluation. Its README organises results into four categories. Reasoning & Knowledge carries GPQA Diamond, CritPt, AA-LCR and HLE-Full. Coding carries DeepSWE, ProgramBench, Terminal-Bench 2.1, FrontierSWE, SWE-Marathon, SciCode and more. Agentic runs to 22 entries — BrowseComp, DeepSearchQA, GDPval-AA v2, Toolathlon-Verified, OSWorld-Verified, τ³-Banking, Legal Research Bench, and others. Vision adds another ten. A full technical report sits in the same repository.

Across all forty-plus named benchmarks, none measures hallucination or factual reliability. The metric that shifted most sharply between generations is the one the vendor’s own materials are silent on — not suppressed, simply never in scope. This is not unique to Moonshot; it is the industry norm. It is also precisely why a buyer reading only vendor tables gets a systematically incomplete picture.

Read those two disclosed behaviours next to the hallucination movement and a coherent profile emerges. A model that answers rather than abstains, and that makes unexpected decisions on the user’s behalf under ambiguity, is the same disposition expressed at two different layers of the stack. Neither is disqualifying. Both argue for tighter scaffolding than you would give a more reticent model — narrower tool permissions, mandatory confirmation steps on writes, and an eval suite that specifically probes ambiguous inputs rather than well-formed ones.

If the hallucination split is the reason to run your own factuality evals, Terminal-Bench 2.1 is the reason to distrust single-number coding claims generally. Three organisations have reported a K3 score on that benchmark. They do not agree, and the disagreement is larger than the gaps that decide most model-selection arguments.

The two figures we would actually stake a decision on are the vendor self-report at 88.3% and Vals AI’s independent replication at 80.9%. That is a 7.4-point gap on a single benchmark, at a single version, for a single model. Vals AI’s broader composite tells a consistent story — it scores K3 at 74.7 on its own index, with 71.6 on CorpFin v2 and 48.9 on MedCode, a set of numbers noticeably more sober than the launch-week framing.

Nothing here implies bad faith. Vendor harnesses are tuned for the vendor’s model: prompt formats, retry policy, tool-call plumbing, timeout budgets and scaffold assumptions all move scores, and a lab that has spent months optimising its own agent loop will naturally extract more from its own weights than a standardised third-party rig does. Harness variance of this size is also not unique to Moonshot — comparable vendor-versus-standardised gaps have been reported for closed frontier models by the same eval-methodology outlet, though we have not re-verified those figures against a first-party source and would not print them as facts.

The practical consequence is simple. If your model-selection argument turns on a gap of under eight points on a coding benchmark, you do not have an argument — you have a harness artefact. That threshold is a useful thing to write into your own cost-aware model routing playbook as a decision rule rather than rediscovering it per release.

The tempting reading of the K3 numbers is that Moonshot traded reliability for scores. The comparator data doesn’t support that reading. The same profile — leading accuracy sitting alongside a hallucination rate that is anything but low — shows up at the top of the same leaderboard today, in a closed frontier model from a lab with very different incentives. To be explicit about what that is and isn’t: it is a single-date snapshot, not a generational comparison. We have no prior-generation AA-Omniscience baseline for Fable 5 and are not claiming its hallucination rate moved in either direction. What the snapshot does show is that an accuracy-led profile is where the top of this board currently sits, which makes the question an evaluation-discipline one rather than a Moonshot-specific flaw.

The Fable 5 row is the load-bearing one, and it is worth being precise about what we can and can’t assert. Artificial Analysis’s own pages confirm that Fable 5 posts the highest AA-Omniscience accuracy on the board at 61% and an Index of 40, and its own commentary attributes that leading position to accuracy rather than to low hallucination. Its per-model hallucination percentage renders inside a client-side chart that we could not read as text, and while several independent outlets relay a specific figure sourced to Artificial Analysis, we are not printing a number we could not verify at source. Directionally: the leader on this benchmark is not the model that guesses least.

That is the finding, and it is a snapshot claim rather than a trend claim. Read the board on one date and the best-scoring knowledge model and the best-scoring open-weights model both present accuracy-led profiles — accuracy is where the training effort went, and abstention discipline is not what the composite rewards. The only generational movement we can actually evidence here is Moonshot’s own, K2.6 to K3; if you were expecting that kind of release-over-release progress to monotonically reduce confident wrong answers, that one comparison already says otherwise. We deliberately aren’t relitigating the broader capability comparison here; our pre-release Fable 5 comparison covers that ground, and our release write-up covers what actually shipped.

Projecting forward: if the composite indices that drive procurement keep rewarding attempt-rate, the rational training response is to attempt more. We would expect the next twelve months to produce models that score better on knowledge composites while getting harder to deploy unsupervised — unless buyers start asking for the sub-metric rather than the index. The correction, if it comes, will come from procurement discipline rather than from model releases.

None of the preceding sections requires a research budget to act on. The gates below are what we run when a model like K3 lands, and they are deliberately cheap — the point is that they are run at all, and that a model can fail one of them without failing the others.

Gate four deserves a note on where K3 actually sits commercially, because open weights invite an assumption of cheapness that the hosted numbers don’t support. At $0.94 blended per task on Artificial Analysis’s Index run, K3 came in below GPT-5.6 Sol at $1.04 and roughly half of Claude Opus 4.8 at $1.80 — but roughly three times GLM-5.2 at $0.32 and more than twenty times DeepSeek V4 Pro at $0.04. It is mid-pack, not the budget option. Throughput reinforces the point: Artificial Analysis measured K3’s output at around 33 tokens per second against roughly 72 for Claude Fable 5 in the same side-by-side, so latency-sensitive surfaces will feel the difference before the invoice does.

The cache-hit economics are the lever worth engineering around. A cache hit costs a tenth of a miss on input tokens, which means prompt architecture — stable system prefixes, deterministic tool schemas, batched context — moves your bill more than model choice does at these volumes. That is the same discipline we apply across production content pipelines regardless of which model sits underneath.

Model selection is per-workload, not per-model. K3’s profile — strong agentic and coding numbers, open weights, mid-pack cost, an elevated tendency to answer rather than abstain — maps cleanly onto some jobs and poorly onto others.

The through-line is that none of these calls came from the Intelligence Index. They came from sub-metrics, vendor disclosures and independent replications — the layer underneath the rank. Any team adopting frontier models at pace needs that layer as standing infrastructure rather than a per-launch scramble, which is the starting point of most of our AI transformation engagements.