Voice AI Benchmarks: How to Read Vendor Claims Right

Voice AI benchmarks have a specific failure mode that text-model leaderboards mostly avoid: two credible parties can publish numbers about the same models, disagree completely, and both be reporting honestly. One measured a full-duplex conversation. The other measured batch transcription. Neither page says so in the headline, and the gap between those two things is where most procurement mistakes get made.

The live example is xAI's Grok Voice Think Fast 2.0 announcement of July 29, 2026, which carries two very different kinds of claim in the same post: a reproduced third-party chart with attributed scores, and a separate accuracy multiplier from xAI's own evaluation harness. Read quickly, they blur into one impression. Read carefully, only one of them is something you can independently check.

This guide separates the two, sets them against Artificial Analysis's independent speech-to-text leaderboard, unpacks what word error rate actually counts, opens up the composite Speech-to-Speech Quality Index, and ends with a harness you can run on your own audio. Every figure below is attributed to the page it came from and labelled with how it was produced.

xAI announced Grok Voice Think Fast 2.0 on July 29, 2026. The post contains a benchmark table with a claimed Artificial Analysis Speech-to-Speech Quality Index of 82.9% for the new model, up from 75.7% for Think Fast 1.0, alongside GPT-Realtime-2.1 (High) at 79.1% and Gemini 3.1 Flash (High) at 69.5%. Those competitor figures are sourced from and attributed to Artificial Analysis rather than generated by xAI — an important detail, because it means the numbers are checkable against a third party.

Elsewhere in the same post sits a claim of an entirely different character. xAI states that in its own evaluation across thousands of short phrases in 24 languages, Grok Voice Think Fast 2.0 demonstrated a 1.5–2.0× improvement in accuracy relative to Deepgram Nova 3 and ElevenLabs Scribe v2, and a 1.4× improvement relative to Think Fast 1.0 — widening to roughly 10× in noisy and telephony-compressed conditions. That evaluation is vendor-run on an unpublished harness. No absolute word error rate is published for any of the three models in that specific comparison, which is what makes it impossible to place on a shared scale.

Both claims can be true at once, and the second one may well be accurate. The problem is that a multiplier without a denominator cannot be ranked. If Nova 3 scores 5.2% word error rate on some test and a competitor is 2× more accurate on that same test, the competitor lands near 2.6% — but nothing in the announcement tells you the harness resembles the leaderboard, so the arithmetic is hypothetical rather than reported. We walked through the launch itself in our earlier look at Grok Voice Think Fast 2.0; this piece is about how to weigh what that announcement said.

For completeness on the commercial side: xAI listed Grok Voice Think Fast 2.0 at $0.08 per minute of audio as an announced API list price on July 29, 2026, and set out a schedule to point the grok-voice-latest alias from Think Fast 1.0 to 2.0 on August 5, 2026. Both are announced positions from that post rather than observed behaviour, and API list prices are the one number in this article that a vendor can change without republishing a benchmark.

Artificial Analysis runs a separate speech-to-text leaderboard that scores transcription models on identical datasets. It is the closest thing this category has to a shared yardstick, and it produces an ordering that a reader of vendor pages alone would not predict.

On the non-streaming index at the time of writing, ElevenLabs Scribe v2 records 2.2% word error rate and Deepgram Nova-3 records 5.2%. That is a gap of 3.0 percentage points, or put as a ratio, Nova-3 makes roughly 2.4 word errors for every one Scribe v2 makes on the same audio. xAI's claim names both products in a single breath, as though they occupy similar ground. On this leaderboard they do not.

Two further numbers on that page are worth carrying with you. A separate xAI-branded product, Grok Speech to Text, appears at 4.0% word error rate — between Nova-3 and Scribe v2, and 2.3 points behind the leaderboard leader. And the overall top result across all 55 evaluated models is Fun-Realtime-ASR-preview at 1.7%, half a point ahead of Scribe v2. Neither figure appears in any vendor announcement discussed here, which is the point: the independent page has a wider field than any single vendor's chosen comparison set.

We could not find a page that puts a full-duplex conversational model's self-reported transcription claim next to an independent transcription leaderboard while keeping the two categories visibly apart. The table below does exactly that. The two groups are deliberately not averaged or reconciled — they measure different pipelines, on different methodologies, and combining them would manufacture a comparison neither source supports.

The gap column is computed, not quoted: each row is the difference between that row’s headline score and the best score inside its own group. Read across the top group and the spread is 13.4 points from leader to laggard. Read across the bottom group and the spread is 3.5 percentage points of word error rate — which sounds smaller until you convert it to a ratio and find the bottom row making roughly three times as many errors as the top row on identical audio. Percentage points and ratios tell different stories about the same two numbers, and vendors reliably quote whichever is more flattering.

Word error rate is a single arithmetic expression: WER = (S + I + D) / N, where S is substitutions, I is insertions, D is deletions, and N is the number of words in the reference transcript. It is Levenshtein edit distance computed at the word level rather than the character or phoneme level. Deepgram’s own methodology page also defines the two metrics that travel with it: word accuracy rate, which is simply 1 − WER, and character error rate, which applies the same counting logic per character.

The formula treats every error identically, and that is the whole problem. Deepgram’s own worked example makes it concrete. Against the reference “can you transfer five thousand dollars to my savings account” — ten words — a system that outputs “can you transfer five hundred dollars my savings account” has committed one substitution (thousand becomes hundred) and one deletion (the word to). Two errors over ten reference words is a 20% word error rate. The arithmetic is correct and the transaction is wrong by an order of magnitude. A generic WER number does not flag that severity at all.

That 2003 finding is more relevant now than when it was published, because almost nothing in a 2026 voice stack consumes a transcript directly. The transcript feeds an agent that extracts an intent, a reference number, an amount, a date. A system with a slightly worse word error rate that never mangles a currency figure is worth more than a lower-WER system that occasionally does — and no leaderboard column captures that. The same distinction between headline accuracy and consequence shows up across model evaluation generally; we explored the reliability side of it in our study of hallucination-rate benchmarks.

The 82.9% headline is not a measurement. It is an aggregate of exactly three component benchmarks: Speech Reasoning, measured on Big Bench Audio; Conversational Dynamics, measured on a Full Duplex Bench subset; and Agentic Performance, measured on τ-Voice. Artificial Analysis is explicit that a model must have results on all three to receive an index score at all — models missing a component are excluded from the index entirely rather than scored as zero, which quietly shapes who appears on the chart in the first place.

Big Bench Audio, the reasoning component, is a 1,000-question dataset adapted from Big Bench Hard, spanning four categories at 250 questions each — Formal Fallacies, Navigate, Object Counting and Web of Lies. The model receives audio input, must generate audio output, and each answer is graded correct or incorrect. That is a narrower thing than “speech reasoning” sounds like.

Now look at where the models actually separate. On the July 29 chart, Grok Voice Think Fast 2.0 leads GPT-Realtime-2.1 on Speech Reasoning by 1.2 points (97.2% against 96.0%) and trails it on Conversational Dynamics by 0.6 points (95.1% against 95.7%). Both models are above 95% on both components. The entire competitive story lives in the third component, where the spread runs from 56.5% down to 37.7%.

Here is the piece of arithmetic that makes the structure legible. For the two models where the July 29 chart lists all three component scores, an equal-weight average reproduces the published index exactly. Think Fast 2.0: 97.2 plus 95.1 plus 56.5 is 248.8, divided by three is 82.93, which rounds to the published 82.9. GPT-Realtime-2.1: 96.0 plus 95.7 plus 45.7 is 237.4, divided by three is 79.13, which rounds to the published 79.1. Two independent checks landing on the decimal is strong evidence that the components carry equal weight in this aggregate, though we would treat that as consistent-with rather than confirmed until the weighting is stated on the methodology page.

If that reading holds, it has a real consequence. A component where every serious model already scores above 95 contributes exactly as much to the headline as the component where models range across nearly 19 points. Two thirds of the index is effectively saturated; one third does all the discriminating. A buyer who cares about conversational feel and a buyer who cares about completing transactions are reading the same 82.9 and should be reading different columns.

That last point deserves an example, because it is where index reading turns into budgeting error. xAI’s announced API list price of $0.08 per minute of audio multiplies out to $4.80 per sixty minutes. It would be natural to set that against a leaderboard’s cost-per-hour-of-input-audio column and treat the two as the same quantity. They are not: the leaderboard figure is derived from a fixed 40-question subset with cached-token discounts and tool calls removed, while a production voice agent bills for output audio and tool use as well. The arithmetic is right and the comparison is still wrong — the same species of mistake we catalogued for coding leaderboards in the SWE-bench Live leaderboard analysis.

τ-Voice is the component doing the discriminating, so it is worth understanding on its own terms. It is a benchmark from Sierra Research, published as an arXiv preprint in March 2026 and announced in plain language on Sierra’s blog on May 1, 2026. It runs 278 customer-service tasks — retail, airline and telecom domains — inherited directly from τ-bench’s text evaluation. That inheritance is the design decision that matters: because the tasks are the same, you can measure what a model loses purely by moving from text to full-duplex voice.

The audio is not clean studio speech. τ-Voice stacks environmental noise mixing (street noise, car horns, background speech not directed at the agent), speaker personas generated across a diverse range, telephony compression using the G.711 μ-law codec at 8 kHz, and packet and frame drops modelled with a Gilbert-Elliott model — on top of natural turn-taking that includes interruptions and backchannels. That is a deliberate reconstruction of a real phone line, not a simulation of a podcast.

Two headline findings come out of it. Voice-agent task completion rose from roughly 30% to roughly 67% between August 2025 and April 2026 — eight months, about 37 percentage points, a little over a doubling. And audio-native models now retain roughly 79% of their text-benchmark task completion when moved to full-duplex voice. Both numbers are more useful than any index score, because the second one quantifies the channel penalty that vendor marketing almost never puts a figure on: about a fifth of a model’s demonstrated text capability does not survive the trip through a phone line.

Those are the three failure modes named in the material available at the time of writing. We are deliberately not putting a total count on the list — the summary sourcing we could verify names three, and a benchmark paper’s taxonomy is exactly the kind of detail that gets miscounted in secondary coverage. If you are building on τ-Voice, read the preprint rather than any summary of it, including this one.

The broader lesson generalises past voice. Benchmarks that inherit their task set from an established text evaluation are far more informative than benchmarks invented alongside the model they first scored, because inheritance gives you a controlled comparison rather than an absolute number floating free. The same argument applies to agent evaluation on desktops and terminals, which we unpacked in our explainer on OSWorld and Terminal-Bench.

There is a genuinely unresolved question sitting in the middle of this comparison, and most coverage papers over it. Artificial Analysis’s speech-to-text leaderboard lists an entry called Grok Speech to Text, attributed to SpaceXAI, at 4.0% word error rate. xAI’s announcement describes Grok Voice Think Fast 2.0, a full-duplex conversational model with built-in transcription, and makes a relative accuracy claim about it.

Whether those are the same weights behind two product names, or two separate product lines, is not established by any primary source we could verify at the time of writing. That matters more than it sounds. If they are the same model, then xAI’s transcription claim has an independent check sitting right there at 4.0% — behind Scribe v2’s 2.2%, which is one of the two products the vendor claim says it beats. If they are different models, the 4.0% figure says nothing whatsoever about Think Fast 2.0 and the vendor claim remains unchecked. The two readings point in opposite directions, so assuming either one is worse than admitting the ambiguity.

The most useful methodology guidance in this whole space is published by a vendor with a stake in the outcome, which is either ironic or reassuring depending on your temperament. Deepgram’s speech-to-text benchmarks page sets out a fair-comparison procedure that any team can run, and it is the right skeleton for an internal evaluation.

Four controls do most of the work. Assemble production-realistic audio from public corpora — LibriSpeech for clean speech, Common Voice for accent variety, TED-LIUM for single-speaker long form — supplemented with your own recordings. Build one identical containerised harness per vendor, so the only variable is the model. Log time-to-first-byte and final latency identically across vendors, because latency comparisons collapse the moment two stopwatches start at different points. Standardise scoring with jiwer, the open-source Python library, to eliminate the formatting and punctuation differences that otherwise move word error rate by more than the model choice does.

To that we would add three checks the vendor page does not cover. Weight your errors by business consequence rather than counting them flat — build a small set of held-out utterances containing the entities that actually cost money in your domain, and score those separately. Test at the noise level your callers actually generate, not at studio quality, because that is where the τ-Voice numbers say capability disappears. And record a retrieval date every time you cite a live leaderboard, because those pages update continuously and a figure without a date is not a citation.

None of this requires a research team. A week of engineering produces a harness that outranks every vendor chart for your specific decision, because it is scored on the audio you actually receive. If you want that built and operated alongside the rest of your measurement stack, our AI and digital transformation engagements start with exactly this kind of comparative evaluation, and our analytics practice handles the instrumentation and reporting layer around it. If your interest is dictation rather than conversational agents, the trade-offs run differently — we compared that field in the open-source voice dictation roundup.

Looking ahead, the pressure in this category runs toward more grounded evaluation, not less. τ-Voice moved the field from clean audio to modelled telephony in a single step, and the capability retention figure it produced is the kind of number that becomes a procurement question once buyers know it exists. Expect the next round of vendor announcements to start quoting channel-degraded results directly — and expect the harnesses behind those results to remain unpublished unless buyers make publication a condition of the conversation.