Laptop workspace for realtime voice engineering

Realtime AI voice systems are judged by more than how quickly the first audio arrives. They must let a person interrupt, correct themselves, pause for a phone call and resume without the assistant repeating audio that the user never heard. If the model’s internal conversation contains a full generated response while the speaker heard only its first half, the next turn can be based on words that never reached the user. This creates confusing answers and makes transcripts unreliable.

1. Model the conversation as explicit states

A robust client should track idle, listening, generating, playing, interrupted, truncating and recovering states. State transitions should be triggered by events rather than scattered booleans. A speech-start event during playback should cancel the current response, stop audio output and reconcile conversation history before a new response is generated. A system alert should pause the audio session without being mistaken for user intent.

Keep one source of truth for the active response identifier and the amount of audio actually played. Associate audio chunks, transcript fragments and cancellation events with that identifier. Ignore late events from a response that has already been cancelled; otherwise delayed audio can appear after the UI has moved on.

2. Distinguish cancellation from truncation

Cancellation stops generation; truncation reconciles what was heard with what the conversation history should contain. If the model generated a long answer but the user interrupted after two seconds, the unplayed portion should not remain as if it had been spoken. A transcript that preserves unheard text can cause the assistant to refer to claims the user never received.

OpenAI’s Realtime conversation guidance describes interruption handling and truncation behavior, including differences between WebRTC and WebSocket clients. In WebRTC and SIP, the server can track the played audio buffer and truncate unplayed output automatically. With WebSockets, the client manages playback and must stop audio and send the appropriate truncation event. Teams should implement the behavior required by their transport instead of assuming the paths are identical.

3. Define event ordering and idempotency

Race conditions appear when speech-start, response-cancelled, audio-buffer-clear and response-created events arrive close together. Assign each user turn a sequence number and make state transitions idempotent. A duplicate cancellation should not create a second recovery branch, and a stale completion event should not reopen a response that has already ended.

Log event IDs, response IDs, turn IDs, playback offsets and state transitions. Do not log raw audio or full transcripts by default when timing metadata is enough to debug the failure. A structured trace can show whether the client stopped playback before truncation, whether the server acknowledged cancellation and whether the next response used corrected history.

4. Treat operating-system interruptions as normal events

Phone calls, Bluetooth route changes, microphone permission changes and system alerts are ordinary mobile conditions. Apple documents audio-session interruption notifications and the need for apps to respond to them. On an interruption, preserve only safe state, stop capture when required and show whether the session paused, ended or can resume.

Do not automatically restart microphone capture merely because an interruption ended. Require the product’s consent and permission model to remain valid. If the audio route changes from earbuds to the device speaker, confirm the state rather than assuming the previous route still exists. In video, camera state should be handled independently from audio state.

5. Recover without duplicating user intent

After a network drop, determine whether the prior turn was committed, partially played or never acknowledged. Retrying a whole turn without a stable identifier can duplicate the user’s request and trigger two model responses. Use an explicit recovery policy: replay the last confirmed state, ask the user to repeat only when necessary, and mark uncertain transcript segments rather than silently treating them as complete.

Track the proportion of interruptions that recover cleanly, duplicate responses, transcript-to-audio mismatch and time to resume useful interaction. Define retry budgets and timeout behavior by failure class rather than using one global retry rule.

6. Test a real interruption matrix

Test a user interrupting at the start, middle and end of a sentence; a system alert during generation; an audio route change during playback; a second interruption while truncation is pending; and a reconnect after cancellation. Repeat with WebRTC and WebSocket transports where supported. Verify that the assistant never resumes speaking unexpectedly after the user ends the session.

Realtime voice feels natural when the system handles the moments that demos often skip. An explicit state machine makes those moments observable and testable, while transport-specific behavior prevents transcript drift and duplicate responses.

7. Add measurable release gates

Track interruption recovery rate, duplicate response rate, transcript-to-audio mismatch, time to resume and the number of sessions that require a restart. Set a release threshold for each metric and keep the same test cases across versions. A change that improves median latency but increases unexpected playback should not automatically pass. Include device and transport coverage in the release record, because a WebSocket client and a WebRTC client can have different responsibilities for truncation. Finally, retain a small set of anonymized traces that demonstrate both a successful interruption and a failure. Those traces help new engineers understand the intended state transitions without exposing full user conversations.

Sources