tvm

Yes. Web-llm is a wrapper of tvmjs: https://github.com/apache/tvm Just wrappers all the way down

This is coming! Myself and others at OctoML and in the TVM community are actively working on multi-gpu support in the compiler and runtime. Here are some of the merged and active PRs on the multi-GPU (multi-device) roadmap: Support in TVM’s graph IR (Relax) - https://github.com/apache/tvm/pull/15447 Support in TVM’s loop IR (TensorIR) - https://github.com/apache/tvm/pull/14862 Distributed dialect of TVM’s graph IR for multi-node (GSPMD-type): https://github.com/apache/tvm/pull/15289 The first target will be LLM's on multiple NVIDIA GPUs but as with all of MLC-LLM effort, the approach will generalize to other hardware including AMD's wonderful hardware.

I was going to answer the same, I find the approach of machine learning compilers that directly compile models to host and device code better than having to bring a huge runtime. There are exciting projects in this area like TVM Unity [1], IREE [2], or torch.export [3] [1] https://github.com/apache/tvm/tree/unity [2] https://github.com/openxla/iree [3] https://pytorch.org/get-started/pytorch-2.0/#inference-and-e...

TVM, similar to IREE, also has a good support for Vulkan. It compiles a tensor DSL written in Python into SPIR-V. AMD is using it in production for running deep learning models on their APU. https://github.com/apache/tvm

You can use apache tvm [1] to get a static runtime. I'm not sure of the state of sparsification acceleration support, but at least quantization is supported. Edit: it's in C++ and has rust bindings [2] [1] https://tvm.apache.org/ [2] https://github.com/apache/tvm/tree/main/rust

Note that TVM (https://github.com/dmlc/tvm) is a similar system (the IR is a fork of the Halide IR, but substantially improved), and is much more suited to deep learning, with tools like: - TOPI, a library of optimized routines for common deep learning operations - Importers for TensorFlow, ONNX (PyTorch/Caffe2), Keras, MXNet, DarkNet and other frameworks. - AutoTVM for automatically finding fast schedules for arbitrary devices, much faster than random search or the automatic schedules generated by e.g. Halide. - Various mobile GPU runtimes (OpenGL, OpenCL, Vulkan, etc), compared to Halide - Large community contributing optimized runtimes (e.g. Intel + Amazon contributing CPU improvements, improved schedules/declarations, etc) Highly recommend checking it out.

Current tally of high-performance, deep-learning-oriented DSLs/IRs/compilers, in no particular order: - TensorComprehensions (Facebook): https://github.com/facebookresearch/TensorComprehensions - XLA (Google): https://www.tensorflow.org/performance/xla/ - taco (MIT): http://tensor-compiler.org/ - DLVM (UIUC): http://dlvm.org/ - nGraph (Intel): http://ngraph.nervanasys.com/docs/cpp/ - TVM (DMLC): https://github.com/dmlc/tvm Honorable mention to Julia (http://julialang.org) as well.

part of TVM https://github.com/dmlc/tvm is built with primitives in Halide. Halide is indeed super cool and TVM benefit a lot from its experience. While Halide optimizes CPU and image processing workload well. TVM also focus specifically on deep learning and offers more optimizations on multi-core, GPU and other hardwares. This requires rethink of quite a lot of designs. NNVM compiler is built on top of TVM, with additional graph level optimizers, and the two forms an end to end pipeline