smart-turn

me improve it. The link takes you to a sample of it running in the browser. There's also an SSM version but that was purely for experimental purposes. While SSMs do well on streaming, there isn't much to gain from it here, since long memory of audio stream isn't that important for turn detection, just the most recent few seconds really, so the GRU version does just fine (or better) and uses much less CPU. More details are in the repo: https://github.com/sshah03/CueLight For those interested in SmartTurn: https://github.com/pipecat-ai/smart-turn

interruptions common with traditional approaches. Source: https://deepgram.com/learn/introducing-flux-conversational-s... So according to them, end-of-turn detection isn't just based on semantic content of the transcript (which makes sense given the latency), but rather the the characteristics of the actual audio waveform itself as well. Which Pipecat (open source voice AI orchestration platform) actually does as well seemingly with their smart-turn native turn detection model as well: https://github.com/pipecat-ai/smart-turn (minus the built-in transcription)

sockets [2] - https://github.com/pipecat-ai/pipecat-flows/ [3] - https://github.com/pipecat-ai/smart-turn Disclaimer: I spend a lot of my time working on Pipecat. Also writing about both voice AI in general and Pipecat in particular. For example: https://voiceaiandvoiceagents.com/

he team at LiveKit released an open weights model that does text-based turn detection. [3] I was really excited to see that, but I'm not super-optimistic that a text-input model will ever be good enough for this task. (A good rule of thumb in deep learning is that you should bet on end-to-end.) So ... I spent Christmas break training several little proof of concept models, and experimenting with generating synthetic audio data. So, so, so much fun. The results were promising enough that I nerd-sniped a few friends and we started working in earnest on this. The model now performs really well on a subset of turn detection tasks. Too well, really. We're overfitting on a not-terribly-broad initial data set of about 8,000 samples. Getting to this point was the initial bar we set for doing a public release and seeing if other people want to get involved in the project. There are lots of ways to contribute. [4] Medium-term goals for the project are: - Support for a wide range of languages - Inference time of <50ms on GPU and <500ms on CPU - Much wider range of speech nuances captured in training data - A completely synthetic training data pipeline. (Maybe?) - Text conditioning of the model, to support "modes" like credit card, telephone number, and address entry. If you're interested in voice AI or in audio model ML engineering, please try the model out and see what you think. I'd love to hear your thoughts and ideas. [1] https://news.ycombinator.com/item?id=40345696 [2] https://x.com/kwindla/status/1870974144831275410 [3] https://blog.livekit.io/using-a-transformer-to-improve-end-o... [4] https://github.com/pipecat-ai/smart-turn#things-to-do