A voice assistant can have excellent recognition accuracy and still feel impossible to talk to. The failure often begins before language understanding: voice activity detection (VAD) closes a turn too early, treats background noise as speech or waits through an awkward silence before answering. Teams need to measure the endpointing decision separately from the model’s response time. This engineering guide defines a test corpus, event instrumentation and acceptance thresholds for live AI conversations. Its numeric targets are example starting hypotheses, not universal performance guarantees.

Distinguish three clocks in every spoken turn

Record the microphone capture timestamp, the estimated start and end of user speech, the VAD decision timestamp and the first audible response sample. These are different events. A model can generate tokens quickly while a conservative endpoint detector adds hundreds of milliseconds of perceived waiting. Conversely, a detector that commits too early creates repeated interruptions and forces users to restate sentences. Keep monotonic timestamps on the client and server, and use trace identifiers to align them without trusting wall-clock synchronization. Report silence-to-decision, decision-to-first-audio and total spoken-turn latency separately. Median latency alone is insufficient; include p95 and p99 where sample size supports them. A good dashboard shows whether a slow conversation was caused by endpointing, network transport, synthesis or an actual model delay.

Build a corpus of realistic utterance boundaries

A validation set should contain short commands, long explanations, hesitations, self-corrections, whispered speech, overlapping speakers and sudden environmental noise. Include names, numerals and languages relevant to actual users. A single clean studio recording will produce misleading thresholds. Label speech start, speech end and intended continuation using two human annotators, then adjudicate disagreements. Include deliberately ambiguous phrases such as “I wanted to… actually, never mind” to expose cases in which silence does not signal completion. A fair experiment runs the same audio against baseline and candidate VAD settings, with no hidden change in speech recognition or model version. Use recordings with consent or synthetic material that does not reveal real user conversations. Report labeling uncertainty rather than pretending that all pauses have one correct interpretation.

Treat transport levels as signals, not declarations of intent

IETF RFC 6464 (December 2011) specifies an RTP header extension that can communicate audio level information. Such indicators may help identify active streams, but amplitude is not the same as whether the speaker has finished a thought. A nearby fan can be loud; a trailing syllable can be quiet. The client should combine appropriate acoustic evidence with explicit user controls such as push-to-talk or tap-to-finish where these make sense. Distinguish a genuine endpoint from a packet loss gap and log which signal produced the decision. Do not promote a transport-level measurement to a psychological interpretation. Keep a fallback that lets the user correct the turn boundary without having to restart the session.

Instrument real call behavior using standard telemetry

The W3C WebRTC Statistics API specification, published as a Candidate Recommendation Draft on 25 September 2025, defines statistics objects available through getStats. It can support investigation of packet loss, jitter and related transport conditions, but VAD decisions and model timing require application instrumentation. Collect coarse audio-level bins rather than raw waveforms for routine metrics when possible. Each event should include session ID, device class, network category, codec and a reason code for endpoint decisions. Protect identifiers and avoid logging transcript text by default. Create a trace view where engineers can distinguish “speech was quiet,” “RTP packets stopped,” “VAD timed out” and “the response generator was late.” These diagnoses call for different fixes.

Evaluate interruption recovery as a separate product behavior

Users interrupt assistants naturally. Test an interruption while the assistant is speaking, a false wake caused by nearby speech, and an interruption that begins just as playback ends. The system should stop playback promptly when the interruption is intentional, but should not destroy the conversation state for every background sound. Define whether a partially delivered assistant answer is stored in memory as complete, partial or canceled. A transcript that shows words never heard by the user creates later confusion. Version the turn state machine and replay recorded events against new code before rollout. Failure categories should include clipped user speech, phantom interruption, unwanted reply continuation and duplicate synthesis. A reduction in average delay is not a win if users must repeat themselves more often.

Run controlled thresholds instead of guessing one universal VAD number

Prepare a table comparing a conservative, balanced and aggressive end-of-speech policy. For each, record early termination rate, false start rate, silent wait, rate of repeated utterances and user-rated turn comfort. The trial must report the hardware, microphone conditions and language mix. A configurable timeout can help specific accessibility needs, but should not silently infer disability from speech style. Keep per-session changes reversible and explain them in ordinary language such as “wait a little longer before replying.” Where voice is unusable, make a clear transition to text. Statistical comparisons should include confidence intervals or at least denominator counts, not a single polished percentage from a handful of test calls.

Use an acceptance gate that protects conversation agency

A release review should listen to failure clips that the team is authorized to access, not only stare at a dashboard. Compare 30 or more representative scripted scenarios per major target environment, then expand the sample before making a strong performance claim. Confirm microphone permission denial, muted device, Bluetooth handoff, network stalls and background interruptions. Keep annotated clips or reproducible synthetic versions as regression fixtures. Require that the client can cancel, retry and switch to text without losing context. Document the largest remaining failure category and who owns it. That evidence creates a more useful engineering contract than the vague promise of “natural real-time AI voice,” because the team can reproduce regressions as models, devices and networks change.

Sources

IETF RFC 6464, Audio Level Indication, December 2011; W3C, WebRTC Statistics API, Candidate Recommendation Draft, September 2025.