---
name: "voice_agent_design"
description: "Design production voice AI agents: set a latency budget across VAD/STT/LLM/TTS, tune turn-taking and barge-in, pick a TTS voice for telephony vs WebRTC, write voice-safe agent prompts, and build a fallback matrix so the agent degrades to text or a human instead of dying. Triggers when someone asks to build a voice agent, cut voice-agent latency, handle interruptions, choose a TTS voice, or plan error recovery for a voice bot."
metadata: { "category": "devtools" }
---

# Voice Agent Design

## Purpose
Turn "voice is text plus a microphone" into a real conversational design: measurable latency targets, sane turn-taking, graceful interruptions, a voice matched to the channel, and fallbacks that keep the session alive when audio fails.

## Workflow

### 1. Choose the architecture
- **Cascaded (STT -> LLM -> TTS)** is the default: text at every boundary gives you logs, swappable providers, moderation hooks, and predictable cost. Pick this for auditability, compliance, tool use, and provider flexibility.
- **Speech-to-speech** only when conversational naturalness and the lowest interruption latency matter most and you can accept opaque failures and token cost that grows with call length.
- Colocate your media server and speech providers in the same regional cloud zone. Cross-continent hops cost 80-120ms per turn for nothing.
- Stream everything, always: STT partial transcripts, LLM token streaming, TTS audio chunks. Streaming turns the sum of stages into roughly the max instead of serializing them.

### 2. Set the latency budget
See `references/latency-budget.md` for the per-stage table. The targets:
- Human turn gaps median ~200ms; under ~700-800ms end-to-end p50 feels human; keep p95 under 1500ms.
- LLM time-to-first-token is usually the long pole. Attack it first (faster model tier for first-token, smaller prompt, token streaming).
- A tool-calling turn costs a second LLM round trip. Budget it and play short acknowledgment filler ("sure, one moment") instead of dead air.
- Measure per-turn, per-stage (VAD, STT final, LLM TTFT, TTS first-audio, voice-to-voice). No histogram, no honesty about speed.
- Use `bin/latency_budget.py` to check a measured budget against the targets.

### 3. Tune turn-taking (endpointing)
- Silence-only endpointing cuts callers off mid-sentence. Prefer semantic end-of-turn detection (acoustic + linguistic completeness), e.g. LiveKit's turn model, OpenAI `semantic_vad`, or a learned turn detector.
- Baseline knobs: min silence delay 0.4-0.5s, max force-end delay 3-4s. Too short (~0.2s) causes false end-of-turn; too long (~1.5s) feels unresponsive. Tune with real audio (pauses, accents, noise), never on clean demos.
- See `references/turn-taking-and-bargein.md` for the full tuning table and the LLM-to-TTS sentence-handoff pattern.

### 4. Handle interruptions (barge-in)
On caller speech while the agent is speaking:
1. Stop TTS playback immediately and drop pending audio in the buffer.
2. Cancel in-flight LLM generation and TTS connection.
3. Checkpoint: keep only what the caller actually heard, discard the rest, and tell the LLM the turn was interrupted so it doesn't repeat.
- Filter false triggers with three layers: latch (one interrupt per utterance), debounce (~400ms for echo/noise bursts), and a minimum interim-transcript threshold so "mm-hm" backchannel doesn't count.
- Put background voice cancellation (e.g. Krisp BVC) upstream of VAD so background voices don't fake interruptions.

### 5. Pick the TTS voice and protocol
- Channel decides the voice: telephony (PSTN/G.711) tops out at 8kHz / 4kHz bandwidth, so pick a voice that survives narrowband and use `ulaw_8000` to avoid transcoding hops. WebRTC/browser gets the full wideband voice.
- Judge vendors on **time-to-first-audio** (when the caller hears speech), not just time-to-first-byte or inference numbers, which vendors measure inconsistently. Voice agents want sub-200ms TTFB from the TTS stage.
- Protocol: WebSocket or HTTP chunking when the text source is a stream of LLM tokens; plain REST fits complete pre-written blocks. If streaming, synthesize the first sentence only and start playback immediately; append the rest behind it.
- Confirm critical slots (names, codes, amounts) explicitly by voice. Callers can't see a screen.

### 6. Write voice-safe agent prompts
Run agent copy through `bin/voice_copy_lint.py`. Rules:
- Keep replies short (under ~30 words unless asked for detail), one question at a time.
- No formatting, ever: no bullets, no numbered lists, no raw digits ("3:00 PM" becomes "three pm").
- Use contractions and natural speech. After a mishear or silence, use natural repair language, not a generic apology loop.
- Stop speaking when interrupted. Offer a human transfer when repeated repair fails.

### 7. Build the fallback matrix
Voice will fail. Define a path for every failure point: mic denied, STT dropped, empty transcript, low confidence, TTS failure, unstable network. See `references/fallback-matrix.md`. Two iron rules:
- Never kill the session on a voice failure. Worst case, continue in text mode with the transcript preserved.
- Never take a consequential action (payment, account change) on a low-confidence transcript. Low confidence narrows the ask or brings a human.

## Output Contract
A voice-agent design deliverable has:
1. **Latency budget table** with p50 targets per stage (VAD/STT/LLM/TTS) and a measured voice-to-voice p50/p95 after the first test call.
2. **Turn-taking config**: endpointing type and tuned delays, with the test audio it was validated against.
3. **Barge-in spec**: stop/clear/cancel/checkpoint sequence plus the three-layer false-trigger filter values.
4. **Voice choice**: TTS provider/model, channel-matched sample rate, streaming protocol, measured time-to-first-audio.
5. **Prompt rules**: voice-safe system prompt section (brevity, no formatting, spelled-out numbers, repair language).
6. **Fallback matrix**: every failure point mapped to detection, caller message, and alternate path. At minimum: mic denied, STT failure, low confidence, TTS failure, network instability, repeated failure.
7. **Verification log**: latency histogram from real calls and a degradation test proving low-confidence transcripts never trigger consequential actions.

## Operating Rules
1. Verify the budget with a histogram from real calls, never vendor marketing numbers.
2. Tune endpointing with real noisy audio, never clean demo recordings.
3. First-audio latency is the only TTS number that matters to the caller; treat other vendor numbers as incomplete.
4. Silence and low confidence and off-script are different failures with different recovery scripts; one generic "sorry, didn't get that" branch is a design failure.
5. Keep the session alive above all: text mode, human handoff, or retry, never a dropped caller.
6. Run both `bin/` helpers before shipping a design; fix what they flag.