willow

significantly better with the ESP-BOX-3 vs Pi, etc because the entire device and software stack is holistically engineered for high quality far-field voice capture. Acoustically tuned enclosure, dual microphones, acoustic echo cancellation, blind source separation, voice activity detection, wake word, etc. As I like to tell people - if you think you're going to slap a random microphone together with a Raspberry Pi and compete with Alexa you have another thing coming. I go into greater detail in this issue[1]. Getting high quality audio is step one. Step two is accurate transcription of that audio. Steps 3 - ? are what you're describing. We know what you said, now what do we do with it? Now that we feel steps one and two are more-or-less nailed steps three and beyond are our next focus. The good news is the final step is of course user feedback and we have that handled with text to speech and output on the BOX-3 LCD display. It's just currently the wide gulf in between these that needs focus. In short, our mouth and ears work well. It's the brain that needs work. [0] - https://www.espressif.com/en/news/ESP32-S3-BOX-3 [1] - https://github.com/toverainc/willow/issues/317#issuecomment-...

s considered it does pretty well. I know you're early (we are too!) but in your demo I counted roughly six seconds of latency between the initial hello and response (and nearly 10 for "tell me a joke"). In terms of conversational voice this feels like an eternity. Again, no shade at all (believe me I understand more than most) but just something I thought I'd add from my decades of experience with humans and voice. This is why we have such heavy emphasis on reducing latency as much as possible. For an idea of just how much we emphasize this you can try our WebRTC demo[1] which can do end-to-end (from click stop record in browser to ASR response) in a few hundred milliseconds (with Whisper large-v2 and beam size 5 - medium/1 is a fraction of that) including internet latency (it's hosted in Chicago, FYI). Running locally with WIS and Willow we see less than 500ms from end of speech (on device VAD) to command execution completion and TTS response with platforms like Home Assistant. Granted this is with GPU so you could call it cheating but a $100 six year old Nvidia Pascal series GPU runs circles around the fastest CPUs for these tasks (STT and TTS - see benchmarks here[2]). Again, kind of cheating but my RTX 3090 at home drops this down to around 200ms - roughly half of that time is Home Assistant. It's my (somewhat controversial) personal opinion that GPUs are more-or-less a requirement (today) for Alexa/Echo competitive responsiveness. Speaking of latency, I've been noticing a trend with Willow users regarding LLMs - they are very neat, cool, and interesting (our inference server[3] supports LLamA based LLMs) but they really aren't the right tool for these kinds of tasks. They have very high memory requirements (relatively speaking), require a lot of compute, and are very slow (again, relatively speaking). They also don't natively support the kinds of API call/response you need for most voice tasks. There are efforts out there to support this with LLMs but frankly I find the overall approach very strange. It seems that LLMs have sucked a lot of oxygen out of the room and people have forgotten (or never heard of) "good old fashioned" NLU/NLP approaches. Have you considered an NLU/NLP engine like Rasa[4]? This is the approach we will be taking to implement this kind of functionality in a flexible and assistant platform/integration agnostic way. By the time you stack up VAD, STT, understanding user intent (while allowing flexible grammar), calling an API, execution, and TTS response latency starts to add up very, very quickly. As one example, for "tell me a joke" Alexa does this in a few hundred milliseconds and I guarantee they're not using an LLM for this task - you can have a couple of hundred jokes to randomly select from with pre-generated TTS responses cached (as one path). Again, this is the approach we are taking to "catch up" with Alexa for all kinds of things from jokes to creating calendar entries, etc. Of course you can still have a catch-all to hand off to LLM for "conversation" but I'm not sure users actually want this for voice. I may be misunderstanding your goals but just a few things I thought I would mention. [0] - https://github.com/toverainc/willow [1] - https://wisng.tovera.io/rtc/ [2] - https://github.com/toverainc/willow-inference-server/tree/wi... [3] - https://github.com/toverainc/willow-inference-server [4] - https://rasa.com/

e more-or-less dumping everything else on core 0. Seems to be working well so far! [0] - https://github.com/toverainc/willow

070 is at least 5x faster than a Threadripper 5955WX. Our goal is to provide an open-source commercial voice assistant equivalent user experience and that is and will be fundamentally impossible for the foreseeable future on CPU. Slight tangent but there are users in the space who seem to be under the impression that they can use their Raspberry Pi for voice assistant/speech recognition. It's not even close to a fair fight but with the same implementation and settings a GTX 1070 is roughly 90x (nearly two orders of magnitude) faster[2] than a Raspberry Pi... Yes, all-in a machine with a GTX 1070 uses and order of magnitude more power (3w vs 30x) than a Raspberry Pi but the power cost in even countries with the most expensive power in the world results in a $2-$3/mo cost difference - which I feel, at least, is a reasonable trade-off considering the dramatic difference in usability (Raspberry Pi is essentially useless - waiting 10-30 seconds for a response makes pulling your phone out faster). [0] - https://github.com/toverainc/willow [1] - https://github.com/toverainc/willow-inference-server [2] - https://github.com/toverainc/willow-inference-server/tree/wi...

ed), etc. It's almost impossible to understand "why CUDA" unless you've fought these battles and spent time with "alternatives". I’ve been active in the space for roughly half a decade and when I look back to my early days I’m amazed what a beginner like me was able to do because of CUDA. I still routinely am. What you’re able to actually accomplish with a $1000 Nvidia card and a few lines with transformers and/or a Docker container is incredible. That said I am really looking forward to Apple stepping it up here - I’ve given up on AMD ever getting it together on GPGPU and Intel (with Arc) is even further behind. The space needs some real competition somewhere. [0] - https://github.com/toverainc/willow [1] - https://github.com/toverainc/willow-inference-server

-possible speech recognition via a variety of means (Willow, WebRTC, POST a file, integration with devices and client applications, etc). Results in hundreds of milliseconds or less for most intended speech tasks. See YouTube WebRTC demo[0]. TTS. Primarily provided for assistant tasks (like Willow!) and visually impaired users. LLM. Optionally pass input through a provided/configured LLM for question answering, chatbot, and assistant tasks. Currently supports LLaMA deriviates with strong preference for Vicuna (I like 13B). Built in support for quantization to int4 to conserve GPU memory. Support for a variety of transports. REST, WebRTC, Web Sockets (primarily for LLM). Performance and memory optimized. Leverages CTranslate2 for Whisper support and AutoGPTQ for LLMs. Willow support. WIS powers the Tovera hosted best-effort example server Willow users enjoy. Support for WebRTC - stream audio in real-time from browsers or WebRTC applications to optimize quality and response time. Heavily optimized for long-running sessions using WebRTC audio track management. Leave your session open for days at a time and have self-hosted ASR transcription within hundreds of milliseconds while conserving network bandwidth and CPU! Support for custom TTS voices. With relatively small audio recordings WIS can create and manage custom TTS voices. See API documentation for more information. Much like the release of Willow[1] last week this is an early release but we had a great response from HN and are looking forward to hearing what everyone thinks! [0] - https://www.youtube.com/watch?v=PxCO5eONqSQ [1] - https://github.com/toverainc/willow

t"). In short, the Espressif frameworks we use /technically/ support the "regular" ESP32 but it's so limited (and the ESP BOX/ESP S3 is so cheap) we're not super interested in supporting it. We're aiming for an end-user experience that's competitive with Echo, Google Home, etc in every possible way - speed, quality, reliability, functionality, and cost. In fact, we want to crush them on all points to where there's no reason left to buy one of them. [0] - https://github.com/toverainc/willow/wiki/