sherpa-onnx

So I use a VAD onnx (Silero [1]) to automatically detect when someone is talking, and then it sends the audio into one of the voice recognition libraries. I originally tried to get away with just Whisper Tiny in the chess game [2], but it performs worse on the kinds of short phrases (knight E4, c takes d5, etc) used to dictate chess notation. Even with hotword-based phrasing and corrections, I found its accuracy on brief inputs noticeably poorer. So I switched over to Sherpa [3] trained on gigaspeech. It’s significantly more accurate, but it also comes with a correspondingly larger memory footprint. Ideally, I would have used just one engine, but I needed a fallback for iOS devices (especially older ones) which can easily OOM. [1] - https://github.com/snakers4/silero-vad [2] - https://shahkur.specr.net [3] - https://github.com/k2-fsa/sher

Understood, you want dictation, not a chatbot. That's a valid and different use case. RCLI is Apple Silicon only today because MetalRT is built on Metal. For Linux, the closest thing to what you're describing would be building a virtual input device on top of Whisper or Parakeet (which RCLI supports as STT backends). Parakeet TDT 0.6B has ~1.9% WER, that's very close to production dictation quality. The missing piece on Linux isn't the model, it's the integration: a daemon that captures mic audio, runs STT with hidden latency (streaming partial results), and injects text as keyboard input. sherpa-onnx (https://github.com/k2-fsa/sherpa-onnx) supports Linux and has streaming STT, it might be the best starting point for what your after. We're focused on Apple Silicon for now but broader platform support is on the roadmap.

FWIW this RCLI is only MIT license but their engine MetalRT is commercial. Not sure the license of their models I guess also not MIT. So IMHO this repo is misleading. Not sure why they decided to reinvent the wheel and write yet another ML engine (MetalRT) which is proprietary. I would most likely bet on CoreML since it have support for ANE (apple NPU) or MLX. Other popular repos for such tasks I would recommend: https://github.com/FluidInference/FluidAudio https://github.com/DePasqualeOrg/mlx-swift-audio https://github.com/Blaizzy/mlx-audio https://github.com/k2-fsa/sherpa-onnx

There are many streaming ASR models based on CTC or RNNT. Look for example at sherpa (https://github.com/k2-fsa/sherpa-onnx), which can run streaming ASR, VAD, diarization, and many more.

I used FFI in the past (C header) but sherpa-onnx is available on pub.dev these days. Look for "sherpa_onnx". You can also point sherpa_onnx in your pubspec.yaml file to a local dir (after cloning the repo somewhere on your file system) or point to a specific git commit hash, and don't forget to specify the path because its not the root of the repo. Here's a link to the dir of the flutter package https://github.com/k2-fsa/sherpa-onnx/tree/master/flutter. The downloads of compatible models can be found at their GitHub Releases but tbh it's a bit of a strange setup IMO. Here's the page for TTS models for example: https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-model... That same "release" (page) gets updated from time to time with new models. Use a bookmark :p - @csukuangfj, thanks for sharing the hard work. Nice to see you here.

Has anyone had any luck with a free, offline, open-source, real-time speech-to-speech translation app on under-powered devices (i.e., older smart phones)? * https://github.com/ictnlp/StreamSpeech * https://github.com/k2-fsa/sherpa-onnx * https://github.com/openai/whisper I'm looking for a simple app that can listen for English, translate into Korean (and other languages), then perform speech synthesis on the translation. Basically, a Babelfish that doesn't stick in the ear. Although real-time would be great, a max 5-second delay is manageable. RTranslator is awkward (couldn't get it to perform speech-to-speech using a single phone). 3PO sprouts errors like dandelions and requires an online connection. Any suggestions?

I wanted to bring together a few projects I've been working with recently, and figured the best way to do so was to make an interactive 3D clone of myself for my own website. This uses my flutter_filament[0] package for rendering (ignore the name, it's no longer Flutter-only), Jaspr[1] (a Dart framework for building reactive UIs, syntax inspired by Flutter), Avaturn[2] for the avatar, Cartesia[3] for the voice cloning, DiffSHEG[4] for the skeletal animations (though scaled down considerably), FaceDiffuser[5] for the facial animation and sherpa-onnx[6] for the speech recognition. [0] https://github.com/nmfisher/flutter_filament [1] https://github.com/schultek/jaspr [2] https://avaturn.me [3] https://cartesia.ai [4] https://github.com/JeremyCJM/DiffSHEG [5] https://github.com/uuembodiedsocialai/FaceDiffuser [6] https://github.com/k2-fsa/she