HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
toverainc
willow-inference-server
Python2023–2025
toverainc / willow-inference-server

willow-inference-server

Open source, local, and self-hosted highly optimized language inference server supporting ASR/STT, TTS, and LLM across WebRTC, REST, and WS

AI & Machine LearningSystems & InfrastructureSecurity & PrivacySelf-hostedLearning & ReferenceNetworking
Explore on GitHubRead Hacker News discussions
2Distinct people
11Mentions
0Explicit recommenders
511GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Open source, local, and self-hosted highly optimized language inference server supporting ASR/STT, TTS, and LLM across WebRTC, REST, and WS

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 6:01:32 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month11 dated mentions

Observed mentions: 2023-05-23 14:22:20 UTC to 2025-03-16 19:16:09 UTC.

01232023-05-01 00:00:00 UTC to 2023-06-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 commentsMay 20232023-06-01 00:00:00 UTC to 2023-07-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2023-07-01 00:00:00 UTC to 2023-08-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsApr 20242024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsMar 2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
May 20232023-05-01 00:00:00 UTC2023-06-01 00:00:00 UTC211
Jun 20232023-06-01 00:00:00 UTC2023-07-01 00:00:00 UTC303
Jul 20232023-07-01 00:00:00 UTC2023-08-01 00:00:00 UTC202
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC101
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC000
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC000
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC101
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC101
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC000
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC000
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC000
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC000
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC000
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC000
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC101

What Hacker News says

2 sources
kkielhofner·May 23, 2023
Thanks! Yes, "realtime multiple" is audio/speech length divided by actual inference time. You got it! The demo video is showing the slowest response times because it is using the highest quality/accuracy settings available with Whisper (large-v2, beam 5). Willow devices use medium 1 by default for comparison and those responses are measured in the 500 milliseconds or less range (again depending on speech length) across a wide variety of new and old CUDA hardware. Some sample benchmarks here[0]. Applications using WIS (including Willow) can provide model settings on a per-request basis to balance quality vs latency depending on the task. [0] - https://github.com/toverainc/willow-inference-server#benchma...
In Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST
kkielhofner·May 23, 2023Submission

Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

Hey HN! Willow Inference Server (WIS) is a focused and highly optimized language inference server implementation. Our goal is to "automagically" enable performant, cost-effective self-hosting of released state of the art/best of breed models to enable speech and language tasks: Primarily targeting CUDA (works on CPU too) with support for low-end (cheap) devices such as the Tesla P4, GTX 1060, and up. Don't worry - it screams on an RTX 4090 too! (See benchmarks on Github). Memory optimized - all three default Whisper (base, medium, large-v2) models loaded simultaneously with TTS support inside of 6GB VRAM. LLM support defaults to int4 quantization (conversion scripts included). ASR/STT + TTS + Vicuna 13B require roughly 18GB VRAM. Less for 7B, of course! ASR. Heavy emphasis - Whisper optimized for very high quality as-close-to-real-time-as
Read the rest
-possible speech recognition via a variety of means (Willow, WebRTC, POST a file, integration with devices and client applications, etc). Results in hundreds of milliseconds or less for most intended speech tasks. See YouTube WebRTC demo[0]. TTS. Primarily provided for assistant tasks (like Willow!) and visually impaired users. LLM. Optionally pass input through a provided/configured LLM for question answering, chatbot, and assistant tasks. Currently supports LLaMA deriviates with strong preference for Vicuna (I like 13B). Built in support for quantization to int4 to conserve GPU memory. Support for a variety of transports. REST, WebRTC, Web Sockets (primarily for LLM). Performance and memory optimized. Leverages CTranslate2 for Whisper support and AutoGPTQ for LLMs. Willow support. WIS powers the Tovera hosted best-effort example server Willow users enjoy. Support for WebRTC - stream audio in real-time from browsers or WebRTC applications to optimize quality and response time. Heavily optimized for long-running sessions using WebRTC audio track management. Leave your session open for days at a time and have self-hosted ASR transcription within hundreds of milliseconds while conserving network bandwidth and CPU! Support for custom TTS voices. With relatively small audio recordings WIS can create and manage custom TTS voices. See API documentation for more information. Much like the release of Willow[1] last week this is an early release but we had a great response from HN and are looking forward to hearing what everyone thinks! [0] - https://www.youtube.com/watch?v=PxCO5eONqSQ [1] - https://github.com/toverainc/willow
13 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.