whisperX

Batching is essentially running multiple instances at once, ie bundling 8 segments and running them simultaneously on the processing unit, but which obviously takes more RAM to do. Notice, however, that if you drop the precision to int8 from fp16, you use basically the same amount of RAM as whisper.cpp yet it completes in a fraction of the time using batching [0]. Yes, if you check their community integrations section on faster-whisper [1], you can see a lot of different CLIs, GUIs, and libraries. I recommend WhisperX [2], it's the most complete CLI so far and has features like diarization which whisper.cpp does not have in a production-ready capacity. [0] https://github.com/SYSTRAN/faster-whisper#benchmark [1] https://github.com/SYSTRAN/faster-whisper#community-integrat... [2] https://github.com/m-bain/whisperX

I've actually done a fair bit of ML work in Elixir, in practice I found: 1) It's generally harder to interface with existing libraries and models (example: whisperX [0] is a library that combines generic whisper speech recognition models with some additional tools like discrete-time-warping to create a transcription with more accurate time stamp alignment - something that was very helpful when generating subtitles. But because most of this logic just lives in the python library, using this in Elixir requires writing a lot more tooling around the existing bumblebee whisper implementation [1]). but, 2) It's way easier to ship models I built and trained entirely with Elixir's ML ecosystem - EXLA, NX, Bumblebee. I trained a few models doing basic visual recognition tasks (detecting scene transitions, credits, title cards, etc), using the ex

Whisper-X does speaker-annotated transcripts nicely. I’ve used it for running multi-hour TTRPG sessions with friends and it worked hassle-free after setup. https://github.com/m-bain/whisperX

For this use case, why not use Whisper to transcribe the audio, and then an LLM to do a second step (summarization or answering questions or whatever)? If you need diarization, you can use something like https://github.com/m-bain/whisperX

whisperx does this all quite well and can be run with `uvx whisperx` https://github.com/m-bain/whisperX

Yes for parakeet, but only comparing benchmark results for canary. Whisper also has severe hallucinations on silence and noise and WhisperX helps a lot, it adds voice activity detection i.e. a model to detect when someone speaks, to filter the input before running whisper. https://github.com/m-bain/whisperX

A few months ago I had to work around a problem like this and the best out there was WhisperX. Not sure about transformer.js support. Link to the repo - https://github.com/m-bain/whisperX

Yep, whisper can do that. You can also try whisperx (https://github.com/m-bain/whisperX) for a possibly better experience with aligning of subtitles to spoken words.

I am building VideoToBe.com - I have found that whisperX works the most reliable. https://github.com/m-bain/whisperX It is built on top of OpenAI Whisper, so speech recognition is good, the transcript gives speaker tags as 'SPEAKER_00' and 'SPEAKER_01' etc. Here is how the transcript may look like https://videotobe.com/play/media/1b02f75a-9503-43aa-8956-d18...

I use whisperfile[1] directly. The whisper-large-v3 model seems good with non-English transcription, which is my main use-case. I am also eyeing whisperX[2], because I want to play some more with speaker diarization. Your use-case seems to be batch transcription, so I'd suggest you go ahead and just use whisperfile, it should work well on an M4 mini, and it also has an HTTP API if you just start it without arguments. If you want more interactivity, I have been using Vibe[3] as an open-source replacement of SuperWhisper[4], but VoiceInk from a sibling comment seems better. Aside: It seems that so many of the mentioned projects use whisper at the core, that it would be interesting to explicitly mark the projects that don't use whisper, so we can have a real fundamental comparison. [1] https://huggingface.co/Mozilla/whisperfile [2] https://g

Yes it's still relevant but I prefer WhisperX for some tasks: https://github.com/m-bain/whisperX

https://github.com/m-bain/whisperX looks promising - I'm hacking away on an always-on transcriber for my notes for later search&recall. It has support for diarization (the speaker detection you're looking for). I'm currently hacking away on a mix of https://github.com/speaches-ai/speaches + https://github.com/ufal/whisper_streaming though - mostly because my laptop doesn't have a decent GPU, I stream the audio to a home server instead. But overall it's pretty simple to do after you wrangle the Python dependencies - all you need is a sink for the text files (for example, create a new file for every Teams meeting, but that's another story...)

You don't need to wait, you can use: https://github.com/m-bain/whisperX right now for STT with timestamps and diarization.

I use whisper and pyannote (https://github.com/m-bain/whisperX), but it is a pain to run locally - I run it on a 4080. This seems to be actually trying to identify the speakers. Not sure what they are doing for that.