RealtimeVoiceChat

I think the best example I've seen of the pipeline approach: ASR(STT) -> LLM -> TTS is still from about 10 months ago in HN https://github.com/KoljaB/RealtimeVoiceChat Just a pity the code is undecipherable :(

Yeah local works really fine. I tried this other tool: https://github.com/KoljaB/RealtimeVoiceChat which allows you to live chat with a (local) LLM. With local whisper and local LLM (8b llama in my case) it works phenomenally and it responds so quickly that it feels like it's interrupting me. Too bad that tool no longer seems to be developed. Looking for something similar. But it's really nice to see what's possible with local models.

How does it compare with https://github.com/KoljaB/RealtimeVoiceChat , which is absent of the benchmark ?

Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a... Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it's the same for every voice) Then change the specific_model="Lasinya" line in audio_module.py into specific_model="some_folder_name". If you change TTS_START_ENGINE to "kokoro" in server.py it's supposed to work, what does happen then? Can you post the log message?

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

This is the system prompt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s... My favorite line: "You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama. Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management. It requires a decent C