web-llm

The Prompt API has some advantages like being a little simpler for some things and some potential to standardize a little bit more in some way, but it looks like from this that it will be delayed unfortunately. However, WebLLM (a library, not actual Web API) https://github.com/mlc-ai/web-llm is more capable and will already work using WebGPU.

Transformers.js (https://huggingface.co/docs/transformers.js/en/index) is this. Some demos (should work in Chrome and Firefox on Windows, or Firefox Nightly on macOS and Linux): https://huggingface.co/spaces/webml-community/llama-3.2-webg... loads a 1.24GB Llama 3.2 q4f16 ONNX build https://huggingface.co/spaces/webml-community/janus-pro-webg... loads a 2.24 GB DeepSeek Janus Pro model which is multi-modal for output - it can respond with generated images in addition to text. https://huggingface.co/blog/embeddinggemma#transformersjs loads 400MB for an EmbeddingGemma demo (embeddings, not LLMs) I've collected a few more of these demos here: https://simonwillison.net/tags/transformers-js/ You can also get this working with web-llm - https://github.com/mlc-ai/web-llm - here's my write-up of a demo that uses that: https://simonwillison.net/20

Apparently it does, though I'm learning about it for the first time in this thread also. Personally, I just run llama.cpp locally in docker-compose with anythingllm for the UI but I can see the appeal of having it all just run in the browser. https://github.com/mlc-ai/web-llm https://github.com/ngxson/wllama

I think what you want is this: https://github.com/mlc-ai/web-llm

Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm

Looks like this is a wrapper around: https://github.com/mlc-ai/web-llm Which has a full web demo: https://chat.webllm.ai/

# LLM in the borwser using WebLLM This is an example of how an LLM can be used safely in the browser without the need for servers or external services. This proves one can use AI in edge computing. This particular notebook requires GPU with atleast 8GB of VRAM. This is an amazing way of using LLMs without installing anything. Works across most modern browsers!!! Credits: [Web LLM Github](https://github.com/mlc-ai/web-llm) Note: The notebook takes 3-4 four minutes to load the LLM for the first time. Next time onwards it will only take 25 seconds as the model is cached. Simple queries are taking 3-5 seconds.

How does it compare to WebLLM (https://github.com/mlc-ai/web-llm)?

This is a wrapper around WebLLM [0] and transformers.js [1]. What exactly are you offering on top of those two libraries? [0]: https://github.com/mlc-ai/web-llm [1]: https://huggingface.co/docs/transformers.js/en/index

There are a couple of WebGPU LLM platforms available that form the building blocks to accomplish this right from the browser, especially since the models are so small. https://github.com/mlc-ai/web-llm https://huggingface.co/docs/transformers.js/en/index You do have to worry about WebGPU compatibility in browsers though. https://caniuse.com/webgpu

I am using hugging face transformer.js for that. https://huggingface.co/docs/transformers.js/en/index You can also take a look at Webllm library. https://github.com/mlc-ai/web-llm In the future, google will probably load Gemini nano into chrome, right now it's in beta - https://developer.chrome.com/docs/ai/built-in

Glad for your review! About the source of the search results, both text and images, they're all from [SearXNG](https://github.com/searxng/searxng/). > SearXNG is a free internet metasearch engine which aggregates results from more than 70 search services. Users are neither tracked nor profiled. SearXNG by itself offers a full-stack platform for you to run searches privately (you can find public instances at <https://searx.space/>, and easily host yourself [via docker](https://github.com/searxng/searxng-docker)). About how they scrape other search engines, it's really simple: HTTP calls and parsing of HTML (for most of them). In MiniSearch, I don't need to save the results by myself. The scrape is done in real-time by SearXNG and passed to MiniSearch, which in turn runs a similarity search and filters out the textual results that don't see

> I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely). I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1]. A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces. However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant. [1] https://github.com/mlc-ai/web-llm