voice_agent_design v1.0.0

name: "voice_agent_design" description: "Design production voice AI agents: set a latency budget across VAD/STT/LLM/TTS, tune turn-taking and barge-in, pick a TTS voice for telephony vs WebRTC, write voice-safe agent prompts, and build a fallback matrix so the agent degrades to text or a human instead of dying. Triggers when someone asks to build a voice agent, cut voice-agent latency, handle interruptions, choose a TTS voice, or plan error recovery for a voice bot." metadata: { "category": "devtools" }

Voice Agent Design

Purpose

Turn "voice is text plus a microphone" into a real conversational design: measurable latency targets, sane turn-taking, graceful interruptions, a voice matched to the channel, and fallbacks that keep the session alive when audio fails.

Workflow

1. Choose the architecture

• Cascaded (STT -> LLM -> TTS) is the default: text at every boundary gives you logs, swappable providers, moderation hooks, and predictable cost. Pick this for auditability, compliance, tool use, and provider flexibility.
• Speech-to-speech only when conversational naturalness and the lowest interruption latency matter most and you can accept opaque failures and token cost that grows with call length.
• Colocate your media server and speech providers in the same regional cloud zone. Cross-continent hops cost 80-120ms per turn for nothing.
• Stream everything, always: STT partial transcripts, LLM token streaming, TTS audio chunks. Streaming turns the sum of stages into roughly the max instead of serializing them.

2. Set the latency budget

See references/latency-budget.md for the per-stage table. The targets:

• Human turn gaps median ~200ms; under ~700-800ms end-to-end p50 feels human; keep p95 under 1500ms.
• LLM time-to-first-token is usually the long pole. Attack it first (faster model tier for first-token, smaller prompt, token streaming).
• A tool-calling turn costs a second LLM round trip. Budget it and play short acknowledgment filler ("sure, one moment") instead of dead air.
• Measure per-turn, per-stage (VAD, STT final, LLM TTFT, TTS first-audio, voice-to-voice). No histogram, no honesty about speed.
• Use bin/latency_budget.py to check a measured budget against the targets.

3. Tune turn-taking (endpointing)

• Silence-only endpointing cuts callers off mid-sentence. Prefer semantic end-of-turn detection (acoustic + linguistic completeness), e.g. LiveKit's turn model, OpenAI semantic_vad, or a learned turn detector.
• Baseline knobs: min silence delay 0.4-0.5s, max force-end delay 3-4s. Too short (~0.2s) causes false end-of-turn; too long (~1.5s) feels unresponsive. Tune with real audio (pauses, accents, noise), never on clean demos.
• See references/turn-taking-and-bargein.md for the full tuning table and the LLM-to-TTS sentence-handoff pattern.

4. Handle interruptions (barge-in)

On caller speech while the agent is speaking:

1. Stop TTS playback immediately and drop pending audio in the buffer.
2. Cancel in-flight LLM generation and TTS connection.
3. Checkpoint: keep only what the caller actually heard, discard the rest, and tell the LLM the turn was interrupted so it doesn't repeat.

• Filter false triggers with three layers: latch (one interrupt per utterance), debounce (~400ms for echo/noise bursts), and a minimum interim-transcript threshold so "mm-hm" backchannel doesn't count.
• Put background voice cancellation (e.g. Krisp BVC) upstream of VAD so background voices don't fake interruptions.

5. Pick the TTS voice and protocol

• Channel decides the voice: telephony (PSTN/G.711) tops out at 8kHz / 4kHz bandwidth, so pick a voice that survives narrowband and use ulaw_8000 to avoid transcoding hops. WebRTC/browser gets the full wideband voice.
• Judge vendors on time-to-first-audio (when the caller hears speech), not just time-to-first-byte or inference numbers, which vendors measure inconsistently. Voice agents want sub-200ms TTFB from the TTS stage.
• Protocol: WebSocket or HTTP chunking when the text source is a stream of LLM tokens; plain REST fits complete pre-written blocks. If streaming, synthesize the first sentence only and start playback immediately; append the rest behind it.
• Confirm critical slots (names, codes, amounts) explicitly by voice. Callers can't see a screen.

6. Write voice-safe agent prompts

Run agent copy through bin/voice_copy_lint.py. Rules:

• Keep replies short (under ~30 words unless asked for detail), one question at a time.
• No formatting, ever: no bullets, no numbered lists, no raw digits ("3:00 PM" becomes "three pm").
• Use contractions and natural speech. After a mishear or silence, use natural repair language, not a generic apology loop.
• Stop speaking when interrupted. Offer a human transfer when repeated repair fails.

7. Build the fallback matrix

Voice will fail. Define a path for every failure point: mic denied, STT dropped, empty transcript, low confidence, TTS failure, unstable network. See references/fallback-matrix.md. Two iron rules:

• Never kill the session on a voice failure. Worst case, continue in text mode with the transcript preserved.
• Never take a consequential action (payment, account change) on a low-confidence transcript. Low confidence narrows the ask or brings a human.

Output Contract

A voice-agent design deliverable has:

1. Latency budget table with p50 targets per stage (VAD/STT/LLM/TTS) and a measured voice-to-voice p50/p95 after the first test call.
2. Turn-taking config: endpointing type and tuned delays, with the test audio it was validated against.
3. Barge-in spec: stop/clear/cancel/checkpoint sequence plus the three-layer false-trigger filter values.
4. Voice choice: TTS provider/model, channel-matched sample rate, streaming protocol, measured time-to-first-audio.
5. Prompt rules: voice-safe system prompt section (brevity, no formatting, spelled-out numbers, repair language).
6. Fallback matrix: every failure point mapped to detection, caller message, and alternate path. At minimum: mic denied, STT failure, low confidence, TTS failure, network instability, repeated failure.
7. Verification log: latency histogram from real calls and a degradation test proving low-confidence transcripts never trigger consequential actions.

Operating Rules

1. Verify the budget with a histogram from real calls, never vendor marketing numbers.
2. Tune endpointing with real noisy audio, never clean demo recordings.
3. First-audio latency is the only TTS number that matters to the caller; treat other vendor numbers as incomplete.
4. Silence and low confidence and off-script are different failures with different recovery scripts; one generic "sorry, didn't get that" branch is a design failure.
5. Keep the session alive above all: text mode, human handoff, or retry, never a dropped caller.
6. Run both bin/ helpers before shipping a design; fix what they flag.