Voice Agent Latency: Which Delays Does Your Demo Hide?

Voice agent latency is not one stopwatch reading. A demo can answer immediately with a polite acknowledgment while the useful result arrives much later. Measure the user’s wait for relevant information, the backend’s time to complete the task, and the time needed to handle an interruption as separate outcomes.

This reference defines a proposed timing worksheet for teams comparing voice applications. It contains no measured vendor scores or universal response-time targets. Its purpose is to make two observations comparable: the same start, the same finish, the same task and a visible record of what happened between them.

Begin with the moment the caller experiences as waiting. Are they waiting for evidence that the agent heard them, an answer to a question, or confirmation that an action succeeded? A single call may contain all three. Give each interval a name and keep its definition beside the result.

Full-duplex applications can talk while tools run, so a sequence drawn as speech, model, tool, speech can be misleading. Some intervals overlap. Adding every component duration may double-count elapsed time. Use timestamps to reconstruct the critical path: the operations that actually delayed the relevant result.

The GPT-Live-1 API guide explains the voice/backend split. The table below is intentionally broader than one API. It identifies observations an evaluator should seek; it does not claim every provider exposes every event.

Each row defines an event, rather than claiming a measured duration. Input events describe what the caller supplied. Backend events describe the work. Delivery events describe how a verified result reached the conversation. Interruption events describe a change that can cross all three.

Use a session identifier and a task revision to associate observations. One spoken request may create several tool operations; one backend result may be offered to the voice layer more than once. A row without a matching identifier is difficult to use in a later comparison.

The GPT-Live session documentation states that transcript start and end values describe intervals on the session timeline. They are not wall-clock times, packet-arrival times or exact word alignments. If a transcript arrives late, its arrival does not identify when the caller stopped speaking.

The same documentation says context-append acknowledgments relate to estimated context injection. They do not prove that the model consumed the whole update or spoke it. A client needs audio playback evidence to establish when the user-facing result became audible.

For a measurement setup, record monotonic time within each process and the mapping used to align processes. Use synchronized recordings or a documented clock-alignment procedure for cross-device comparisons. If alignment uncertainty is larger than the difference between two systems, do not claim the smaller difference is meaningful.

Proposed first-audio delay is the first assistant playback timestamp minus the annotated end of the user request. Proposed result delay uses the start of a verified substantive result instead. For an interrupted response, also record whether the important content finished playing; beginning a confirmation is not the same as delivering it.

Backend queue time is tool start minus delegation dispatch only when that delegation directly schedules the tool being measured. If model reasoning or other work happens in between, name the interval dispatch-to-tool-start instead of calling all of it queue time. Labels should reflect the path you can actually observe.

The W3C WebRTC statistics specification defines media and transport measurements, including jitter-buffer delay. These can help diagnose audio delivery. A transport statistic is not a substitute for a speech-to-result measurement: the backend may still dominate the caller’s wait.

When assistant speech begins before the annotated request ends, retain the signed overlap and inspect it. It might be a natural acknowledgment or an unwanted interruption. Clamping negative intervals to zero would hide the very behavior the test needs to explain.

Hold the requested outcome, backend tools, relevant data and model settings stable when testing a voice change. Then vary one condition deliberately: background noise, a long pause, an identifier spoken with a correction, or a delayed tool. Record the condition alongside every run.

Report the distribution of valid intervals and the number of eligible requests. Show timeouts, missing observations and task failures separately. An answer that arrives quickly but contains the wrong result cannot be counted as a successful fast answer. If slow requests are excluded, state the exclusion and its effect on the denominator.

Our voice benchmark interpretation guide covers claims drawn from vendor tests. Here, keep the object local: what did this application do on this task? The background-tool interaction guide helps diagnose cases where useful backend work finishes but the spoken update is stale.

Download the blank latency worksheet. It contains field names for event timestamps, clock domain, operation identity and outcome. Leave unavailable values empty and explain why. Do not fill gaps with inferred timestamps simply to complete a chart.

For every reported improvement, retain enough information to replay the interpretation: the input recording, relevant output audio, event log, test configuration and inclusion rule. Capture only material you are authorized to retain, and use access controls appropriate to the recordings.

Choose a response-time objective only after observing the task and discussing what users can tolerate. A short lookup and a document-generation request have different waiting patterns. Our AI transformation service can help turn those requirements into a scoped evaluation rather than an arbitrary speed target.