Voice AI Benchmarks and Listener Votes Disagree: 14 Models

Artificial Analysis publishes two ways of ranking realtime voice systems, the models that listen and talk back without a separate transcription step. One is an arena: listeners hear two systems answer the same prompt and vote, producing an Elo score. The other is a benchmark set: a speech-reasoning test, a turn-taking test, a time-to-first-audio measurement and a cost per hour. Read on September 25, 2026, the two boards disagree about which systems are good, and the disagreement is not noise.

Of the systems on both boards, the one with the best benchmark scores, Qwen Audio 3.0 Realtime Plus, ranks last on preference, 26th of 26. OpenAI's GPT-Realtime-2.1 outscores the older GPT-Realtime-1.5 on Big Bench Audio (95.8% against 81.4%), ties it on Full Duplex Bench and loses to it on preference. This post joins the two tables, says what each one measures, marks where the confidence intervals make a ranking meaningless, and ends with how to choose a model for a voice agent when the boards point in different directions.

Both columns are from Artificial Analysis' speech-to-speech page, read on September 25, 2026. The preference leader scores 71.3% on Big Bench Audio and 72.3% on Full Duplex Bench, and the arena system with the best benchmark scores has the lowest preference score. That is the whole problem: a buyer reading either board alone would choose a system the other board says is the wrong one.

The arena anchors GPT-Realtime-1.5 at 1000 and scores every other system relative to it. The interval column is the 95% confidence interval Artificial Analysis publishes; where two intervals overlap, the two systems are not distinguishable on preference. Systems labelled cascaded run a separate transcription, text model and speech step rather than one native model, which Artificial Analysis marks on the board and we carry over. Rank numbers are the board's own and skip where our read of the board did not capture a row.

Read the intervals before the ranks. Positions one to five span 1048 to 1096 and every adjacent pair overlaps, so the honest statement is that five systems share the top of the board, and the sixth and seventh overlap the fifth. The one gap that is unmistakable is Qwen's: its upper bound of 819 sits below the lower bound of every other row in this table, though three uncaptured systems at ranks 23 to 25 overlap it. Task completion tells a different story again. The system listeners prefer most completes fewer tasks than ten of the thirteen systems below it, and Grok Voice Think Fast 2.0 High, seventh on preference, has the best completion rate in the table. Our post on Grok Voice Think Fast 2 covers what xAI built it for.

Big Bench Audio tests whether the system reasons correctly about what it heard. Full Duplex Bench tests turn-taking: whether the system yields, interrupts and resumes the way a conversation needs. Time to first audio is the seconds until the system starts speaking. The cost column is Artificial Analysis' figure for an hour of input audio on the vendor's list price. Scores are as published; a dash means the page lists no value.

Two rows are worth a second look. Gemini 3.8 Live scores 91.7% on reasoning, 96.1% on turn-taking, starts speaking in 1.18 seconds, ranks second on preference with the second-best task completion, and costs $0.84 an hour, less than a quarter of the next cheapest listed system. And StepAudio 3 Realtime, top on both benchmarks at 99.7% and 98.9%, takes 8.83 seconds to begin speaking, which disqualifies it from a live conversation whatever it scores. Our post on Gemini 3.8 Live and its extended-thinking variant explains the split between the two Google rows.

The two boards measure different things and neither is wrong. A benchmark scores an answer against a key: did the system reason correctly about the audio, did it yield the turn when it should. A listener votes on an experience: did it sound like a person, did the voice fit, did it feel quick, did it say too much. A system can be correct and unpleasant, or pleasant and wrong, and the boards show both cases. The turn-taking benchmark rewards a system that interrupts at exactly the right moment; a listener may prefer one that waits a beat too long and sounds polite.

The arena also has a sample problem the intervals make visible. Qwen's 171 votes produce an interval 88 points wide; the top-ranked system's 841 votes produce one 55 points wide. A new system enters with a wide interval and a small vote count, and its position moves as votes arrive. That is why we quote the interval with every Elo, why a difference inside overlapping intervals is not a ranking, and why the board should be re-read rather than remembered.

Cascaded systems sit on both boards with their own trade-off. A pipeline of transcription, text model and speech synthesis can reason with a stronger text model than any native voice model carries, and it pays for that in latency and in turn-taking, because the pieces do not share a clock. Our reference on voice-agent latency measures defines time to first audio and the other timings the benchmark board uses.

Prices for the vendor-side comparison are in our text-to-speech ranking for the synthesis half of a cascaded stack and in our post on Gemini 3.8 Flash TTS for the January price change. Our AI transformation practice runs the 50-call test as the first week of any voice-agent build.