web-llm

Transformers.js (https://huggingface.co/docs/transformers.js/en/index) is this. Some demos (should work in Chrome and Firefox on Windows, or Firefox Nightly on macOS and Linux): https://huggingface.co/spaces/webml-community/llama-3.2-webg... loads a 1.24GB Llama 3.2 q4f16 ONNX build https://huggingface.co/spaces/webml-community/janus-pro-webg... loads a 2.24 GB DeepSeek Janus Pro model which is multi-modal for output - it can respond with generated images in addition to text. https://huggingface.co/blog/embeddinggemma#transformersjs loads 400MB for an EmbeddingGemma demo (embeddings, not LLMs) I've collected a few more of these demos here: https://simonwillison.net/tags/transformers-js/ You can also get this working with web-llm - https://github.com/mlc-ai/web-llm - here's my write-up of a demo that uses that: https://simonwillison.net/20

Apparently it does, though I'm learning about it for the first time in this thread also. Personally, I just run llama.cpp locally in docker-compose with anythingllm for the UI but I can see the appeal of having it all just run in the browser. https://github.com/mlc-ai/web-llm https://github.com/ngxson/wllama

I think what you want is this: https://github.com/mlc-ai/web-llm

Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm

Looks like this is a wrapper around: https://github.com/mlc-ai/web-llm Which has a full web demo: https://chat.webllm.ai/

# LLM in the borwser using WebLLM This is an example of how an LLM can be used safely in the browser without the need for servers or external services. This proves one can use AI in edge computing. This particular notebook requires GPU with atleast 8GB of VRAM. This is an amazing way of using LLMs without installing anything. Works across most modern browsers!!! Credits: [Web LLM Github](https://github.com/mlc-ai/web-llm) Note: The notebook takes 3-4 four minutes to load the LLM for the first time. Next time onwards it will only take 25 seconds as the model is cached. Simple queries are taking 3-5 seconds.

How does it compare to WebLLM (https://github.com/mlc-ai/web-llm)?

This is a wrapper around WebLLM [0] and transformers.js [1]. What exactly are you offering on top of those two libraries? [0]: https://github.com/mlc-ai/web-llm [1]: https://huggingface.co/docs/transformers.js/en/index

There are a couple of WebGPU LLM platforms available that form the building blocks to accomplish this right from the browser, especially since the models are so small. https://github.com/mlc-ai/web-llm https://huggingface.co/docs/transformers.js/en/index You do have to worry about WebGPU compatibility in browsers though. https://caniuse.com/webgpu