agents

he audio signal switches from speech to silence and then triggers an end of turn once a configurable amount of silence has transpired. One obvious delta between VAD and how humans do turn detection is we also consider the content of speech (i.e. what someone says). These past few months, we’ve been working on an open weights, content-aware turn detection model for voice AI applications. It was fine-tuned from SmolLM v2 on text, runs on CPU (currently takes 50ms for inference), and uses speech transcriptions as input to predict when a user has completed a thought (also called an “utterance”). Since it was trained on text, notably it works well for pipeline-based architectures (i.e. STT ⇒ LLM ⇒ TTS). We use this model together with VAD to make better predictions about whether a user is done speaking. Here’s some demos -- - Podcast interview: https://youtu.be/EYDrSSEP0h0 - Ordering food: https://youtu.be/fcr8Y-3c4E0 - Providing shipping address: https://youtu.be/2pQWvd6xozw - Customer support: https://youtu.be/YoSRg3ORKtQ In our testing we’ve found: - 85% reduction in unintentional interruptions - 3% false positives (where the user is done speaking, but the model thinks they aren’t) In practice, we still have work to do. We currently delay inference if the model predicts a < 15% chance the user is done speaking. This threshold misses a bunch of middle-of-the-pack probabilities. Next steps are improving the model accuracy, tuning performance, and expanding to support more languages (only supports English rn). Separately, we’re starting to explore an audio-based model that considers not just what someone says but how they say it, which can be used with natively multimodal models like GPT-4o that directly process and generate audio. Code here: https://github.com/livekit/agents/tree/main/livekit-plugins/... Let us know what you think!

ecause websocket isn’t the best choice for client-server communication. The vast majority of packet loss occurs between a server and client device and websocket doesn’t provide programmatic control or intervention in lossy network environments like WiFi or cellular. Packet loss leads to higher latency and choppy or garbled audio.