AI Voice Agents in Production: Latency, Barge-In and Turn-Taking

A chat agent can take four seconds to answer; the typing indicator covers it. On a call, four seconds of silence reads as a dropped connection and the caller says "hello?" — which the agent may treat as a new turn.

Voice has three constraints no other agent interface has:

• The caller's patience is the SLA. There is no scrollback and no retry button.
• There is one channel. No links, no buttons. If the agent cannot say it, it does not exist.
• Errors are audible. A misheard order number is not a quiet wrong answer; it is a confident wrong answer, read back.

That is why voice projects that look excellent in a scripted demo collapse on real traffic. The model is not the variable. The plumbing is.

A voice turn is a chain: endpointing, speech recognition, the model's first token, speech synthesis, then network jitter on both ends. Every stage adds, and the budget is what the caller experiences.

We design to first audio well under a second, with endpointing and partial recognition in the low hundreds of milliseconds, first token in the low hundreds, and audio starting shortly after. Judge the build on p95, not the average — the slow calls are the ones people remember.

Where teams lose the budget:

• Waiting for a final transcript. Streaming partials and speculating on intent buys back more than a model upgrade.
• Non-streaming generation. If the agent waits for a complete response before speaking, the whole budget went to silence.
• Synthesising whole paragraphs. Chunk it: audio starts at the first clause and renders while the caller listens.
• Distance. Recognition, model and synthesis in three regions means three network crossings per turn. Co-locate them.

Measure per stage, per call, and keep the traces. "It felt slow" is not a fixable bug report.

Callers interrupt, and that breaks voice agents in one of two directions.

The first is no barge-in: the caller says "no, I said Thursday", the agent reads its confirmation script, and the call becomes a fight. The second is naive barge-in: any noise stops the agent, so every "mm-hmm" cuts it off mid-sentence.

The workable middle:

• Separate backchannels from interruptions. "Mm-hmm" and "yeah" are acknowledgements, not turn-taking.
• Stop fast, resume gracefully. Keep the interrupted content so the agent picks it up instead of restarting.
• Tune endpointing per language and accent. Too short and it talks over people who pause mid-sentence; too long and it feels dead.
• Handle echo. Speakerphone callers echo their own audio; without cancellation the agent hears itself and interrupts itself.

Turn-taking is also a state problem. If the transcript is stale when the caller changes their mind, the agent answers the previous question — confidently, which is worse than not answering.

Listen to fifty real calls and the same problems appear.

1. It repeats the question. Low recognition confidence with no repair strategy. A repair should rephrase, not repeat. 2. It answers something else. A turn-ordering error. The fix is state, not a better prompt. 3. Dead air after a long answer. Whole-response synthesis. Chunk it. 4. A confirmation loop. Prompt-only control has no state machine to mark a slot filled. Give it one. 5. It never escalates. Or it escalates and drops the context, so the caller repeats everything to a human — worse than no handoff, because it fails twice.

The failures that cost money are entities: order numbers, phone numbers, emails, amounts, surnames. Recognition is good at sentences and fallible at digit strings.

• Read back with structure. Group digits the way people say them; let the caller correct one group, not the whole number.
• Capture slots, not paragraphs. Store the confirmed order number as a field so a later step cannot paraphrase it wrongly.
• Offer a deterministic fallback. Keypad entry or an SMS link for anything long.
• Spell names letter by letter, with a phonetic alphabet, then read it back.
• Test with real accents, on real phones, in real noise — including code-switching, which is normal in the Gulf and India. Machine-translating an English script into Arabic produces an agent that is grammatically correct and socially wrong.

• Containment and escalation by intent — not one blended rate.
• Task completion, not call duration. Short calls can mean efficiency or failure.
• Latency p50 and p95 per stage.
• Repeat and repair rate — and whether the repair worked.
• Handoff quality — did the human get transcript, confirmed slots and a reason?
• Cost per resolved call against the human baseline it replaced.
• A weekly human sample. Fifty calls, listened to by someone who can tell a good call from a fast one.

Off-the-shelf voice platforms fit standard intents: hours, locations, rescheduling, simple qualification. They fit badly when the call has to check inventory, change an order or write to a system of record — that is integration work wearing a voice interface.

Either way, keep the dialogue policy, the integration layer and the call data. Keep recognition and synthesis swappable; that market moves faster than your roadmap.

• Week one. Twenty real call scenarios from actual recordings; run the current agent against them and keep the audio.
• Week two. Instrument latency per stage and log turn-taking events. Fix the three worst.
• Week three. Entity capture, repair phrasing, escalation paths. A hundred scripted calls with people who were not on the project.
• Week four. One intent behind a real number with a human fallback. Listen to fifty calls. Decide on evidence.

Voice is unforgiving, but it is measurable. Almost every "the AI sounded robotic" complaint resolves to a budget, a turn-taking rule or a handoff — engineering decisions, not model choices.