mlc-llm

Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm

It depends on what you mean by "this." MLC's catch is that you need to define/compile models for it with TVM. Here is the list of supported model architectures: https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/m... llama.cpp has a much bigger supported model list, as does vLLM and of course PyTorch/HF transformers covers everything else, all of which work w/ ROCm on RDNA3 w/o too much fuss these days. For inference, the biggest caveat is that Flash Attention is only an aotriton implementation, which besides being less performant sometimes, also doesn't support SWA. For CDNA there is a better CK-based version of FA, but CK doesn't not have RDNA support. There are a couple people at AMD apparently working on native FlexAttention, os I guess we'll how that turns out. (Note the recent SemiAccurate piece was on training, which I'd ag

there's a pretty cool Vulkan LLM engine here for example: https://github.com/mlc-ai/mlc-llm

I got this project[0] running on a Pixel. Looks like it works on some iPhones/iPads as well. [0] https://github.com/mlc-ai/mlc-llm

Machine Learning Compilation (MLC) now supports compiling LLMs to multiple GPUs. For Llama2-70B, it runs 4-bit quantized Llama2-70B at: - 34.5 tok/sec on two NVIDIA RTX 4090 at $3k - 29.9 tok/sec on two AMD Radeon 7900XTX at $2k - Also it is scales well with 8 A10G/A100 GPUs in our experiment. Details: - Blog post: https://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infer... - Project: https://github.com/mlc-ai/mlc-llm

For LLM inference, a shoutout to MLC LLM, which runs LLM models on basically any API that's widely available: https://github.com/mlc-ai/mlc-llm

One of your problems might be that gfx1032 is not supported by AMD's ROCm packages, which has a laughably short list of supported hardware: https://rocm.docs.amd.com/en/latest/release/gpu_os_support.h... The normal workaround is to assign the closest architecture, eg gfx1030, so `HSA_OVERRIDE_GFX_VERSION=10.3.0` might help Also, it looks like some of your tested projects are OpenCL? For me, I do something like: `yay -S rocm-hip-sdk rocm-ml-sdk rocm-opencl-sdk` to cover all the bases. My recent interest has been LLMs and this is my general step by step for those (llama.cpp, exllama) for those interested: https://llm-tracker.info/books/howto-guides/page/amd-gpus I didn't port the docs back in, but also here's a step-by-step w/ my adventures getting TVM/MLC working w/ an APU: https://github.com/mlc-ai/mlc-llm/issues/787 From my experience, R

Maybe they're talking about https://github.com/mlc-ai/mlc-llm which is used for web-llm (https://github.com/mlc-ai/web-llm)? Seems to be using TVM.

you already have TVM for the cross platform stuff see https://tvm.apache.org/docs/how_to/deploy/android.html or https://octoml.ai/blog/using-swift-and-apache-tvm-to-develop... or https://github.com/mlc-ai/mlc-llm

Another Engine: - https://github.com/mlc-ai/mlc-llm

Things are already possible on today's hardware, see https://github.com/mlc-ai/mlc-llm which allows many models to be run on M1/M2 Macs, WASM, iOS and more. The main limiting factor will be small enough, high quality enough models that performance is high enough ultimately this is HW limited and they will need to improve the neural engine/map more computation on to it to make the mobile exp. possible.

With any luck projects like MLC will help close the gap https://github.com/mlc-ai/mlc-llm

In the wild, people tend to use GTPQ quantization for pure GPU inference: https://github.com/PanQiWei/AutoGPTQ And ggml's quant for CPU inference with some offload, which just got updated to a more GPTQ-like method days ago: https://github.com/ggerganov/llama.cpp/pull/1684 Some other runtimes like Apache TVM also have their own quant implementations: https://github.com/mlc-ai/mlc-llm For training, 4-bit bitsandbytes is SOTA, as far as I know: https://huggingface.co/blog/4bit-transformers-bitsandbytes TBH I'm not sure why this November paper on 8 bit is being linked now. Few are running 8 bit model inference when they could fit a better 3-5 bit model in the same memory pool.

We don't necessarily know... Hippo is closed source for now. Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely, but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!