wav2letter

develops these frameworks). Wav2Letter (https://github.com/facebookresearch/wav2letter), the tool by Facebook. These are probably just the most well known. There are many others as well. DeepSpeech is inferior to all the ones above, but maybe simpler. Important is also the data to train these, but you will find quite some resources online for English, e.g. Tedlium, Librispeech, etc. You will find lots of resources actually for ASR. Some random links: https://github.com/gooofy/zamia-speech https://commonvoice.mozilla.org/en/datasets https://www.openslr.org/resources.php To add: If you want to do sth like Descript, you are mostly also interested in accurate time-stamps of the recognized text (start and end time of each spoken word). The end-to-end models are usually not so good at this (the goals is mostly to get a good word-error-rate (WER)). The conventional hybrid NN-HMM is maybe actually a better choice for this task.

thub.com/NVIDIA/OpenSeq2Seq). * Facebook Fairseq (https://github.com/pytorch/fairseq). Attention-based encoder-decoder models mostly. * Facebook wav2letter (https://github.com/facebookresearch/wav2letter). ASG model/training. * Vosk (https://github.com/alphacep/vosk-api). Offline lightweight speech recognition API with support for 10 languages. And there are much more.

open source release of Googles internal ASR system, and used by Google in production (their internal version of it, which is not too much different). * NVIDIA OpenSeq2Seq (https://github.com/NVIDIA/OpenSeq2Seq). * Facebook Fairseq (https://github.com/pytorch/fairseq). Attention-based encoder-decoder models mostly. * Facebook wav2letter (https://github.com/facebookresearch/wav2letter). ASG model/training. * (RETURNN (https://github.com/rwth-i6/returnn) and RASR (https://github.com/rwth-i6/rasr), our own, although this is currently free for academic use only. It is used in production as well. Supports hybrid NN-HMM, CTC, end-to-end attention-based encoder-decoder, transducer, etc.) And there are much more. You will also find lots of ready-to-use trained models.

ot of architectures, the most accurate of which was a transformer model trained with attention and seq2seq criterion. However in my experience it was very large and slow to train. 3. Using Byte-Pair Encoding Streaming ConvNets model uses 10,000 word pieces, which is a similar thought (the token set FB uses is here: [4] but I've had better luck with a sentencepiece model trained on more text data) instead of a ~26 letter alphabet from the original conv_glu model. 4. Better Encoder The blog post conclusion here isn't very obvious, it seems to hand wave that you could maybe use a transformer architecture but doesn't go into specifics. 5. Balance Capacity I don't completely understand what they changed here but it seems useful. 6. Stabilize the Training in Different Domains, Balance Generalization When adding domains to wav2letter, I've basically trained entirely from scratch using all available data. I long for a catastrophic forgetting mitigation like EWC but for now I've had a lot of success by just making the data funnel bigger and more varied. Data matters quite a bit, in this post [5] I compare a model I trained on a wide variety of data, with Facebook's equivalent model as well as Facebook's State Of The Art model. The wide-variety model generalized significantly better than both of Facebook's examples that had only trained on audiobooks. 7. Make a very fast decoder. The beam search decoder in this branch [6] of Talon's wav2letter fork was at the time significantly faster than Facebook's decoder. It was hand optimized and even includes some code for threaded decoding (which I'm not using at the moment because single threaded is fast enough for my workload). On single threaded decoding it can hit a realtime factor in the ballpark of 0.01x, or 100 seconds of audio per cpu second. With my smallest model (a significantly reduced size conv_glu wav2letter model), which is also in the 0.01x ballpark for encoding, we can consistently hit around 0.02x end to end on a consumer CPU. To demonstrate, here is a random sample of debug timing information from my machine when recently decoding command phrases with wav2letter during real interactive use (the very long audio samples aren't the model's fault, I was testing a mic in a noisy environment): [audio]=1260.000ms [emit]=16.164ms (0.01x) [decode]=11.326ms (0.01x) [total]=27.490ms (0.02x) [audio]=1260.000ms [emit]=16.164ms (0.01x) [decode]=11.326ms (0.01x) [total]=27.490ms (0.02x) [audio]=930.000ms [emit]=12.853ms (0.01x) [decode]=3.716ms (0.00x) [total]=16.569ms (0.02x) [audio]=1080.000ms [emit]=14.936ms (0.01x) [decode]=3.296ms (0.00x) [total]=18.231ms (0.02x) [audio]=1020.000ms [emit]=12.976ms (0.01x) [decode]=10.058ms (0.01x) [total]=23.034ms (0.02x) [audio]=930.000ms [emit]=13.836ms (0.01x) [decode]=9.998ms (0.01x) [total]=23.834ms (0.03x) [audio]=1140.000ms [emit]=14.535ms (0.01x) [decode]=3.589ms (0.00x) [total]=18.124ms (0.02x) [audio]=1650.000ms [emit]=20.516ms (0.01x) [decode]=5.281ms (0.00x) [total]=25.797ms (0.02x) [audio]=3480.000ms [emit]=31.386ms (0.01x) [decode]=32.966ms (0.01x) [total]=64.352ms (0.02x) [audio]=29760.000ms [emit]=253.671ms (0.01x) [decode]=325.198ms (0.01x) [total]=578.869ms (0.02x) [audio]=8850.000ms [emit]=70.844ms (0.01x) [decode]=76.267ms (0.01x) [total]=147.111ms (0.02x) [audio]=960.000ms [emit]=13.924ms (0.01x) [decode]=8.135ms (0.01x) [total]=22.060ms (0.02x) [audio]=1770.000ms [emit]=19.760ms (0.01x) [decode]=18.612ms (0.01x) [total]=38.372ms (0.02x) [audio]=930.000ms [emit]=12.488ms (0.01x) [decode]=6.842ms (0.01x) [total]=19.330ms (0.02x) [audio]=1050.000ms [emit]=15.726ms (0.01x) [decode]=7.645ms (0.01x) [total]=23.371ms (0.02x) [audio]=930.000ms [emit]=13.899ms (0.01x) [decode]=8.714ms (0.01x) [total]=22.613ms (0.02x) [audio]=40410.000ms [emit]=292.916ms (0.01x) [decode]=289.109ms (0.01x) [total]=582.025ms (0.01x) [1] https://github.com/facebookresearch/wav2letter/tree/master/r... [2] https://github.com/facebookresearch/wav2letter/issues/543#is... [3] https://github.com/facebookresearch/wav2letter/tree/master/r... [4] https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets... [5] https://github.com/facebookresearch/wav2letter/issues/577 [6] https://github.com/talonvoice/wav2letter/tree/decoder

t their best numbers both with and without that step (as you said “if kaldi is trained on the same data”) Without 60k unsupervised audio: test-clean: 2.31% WER test-other: 5.18% WER With 60k audio: test-clean: 2.03% WER test-other: 4.11% WER That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model: Without 60k or language model: (test-clean / test-other WER) 3.05 / 7.01 With 60k without language model: 2.30 / 5.29 Am I missing a kaldi whitepaper here? I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.) If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.