Reading up on latency

Andrew Hunter contrasts hyperscale performance tuning, where tiny changes yield big savings, with latency‑focused trading systems where profiling must capture rare, critical moments.

A developer built a sub-500ms latency voice agent by co-locating and aggressively pipelining STT, LLM, and TTS stages. Commenters debate trade-offs vs. end-to-end models, endpoint detection alternatives, and the difficulty of human-like turn-taking.

By piping Deepgram Flux, Groq's LLM, and ElevenLabs TTS with geographic optimization, the author achieved ~400ms end-to-end voice agent latency, beating Vapi by 2x.

Prompt caching reduces latency by up to 80% and costs by up to 90% by reusing computed key/value tensors for repeated prompt prefixes in OpenAI's API.