candle

Integration with candle is already available! https://github.com/huggingface/candle/pull/3934/

Since NVIDIA owns huggingface now and huggingface has the excellent Candle [1] crate for inference on Rust, this seems like a good step towards nice native Rust kernels. [1] https://github.com/huggingface/candle

Not similar in the way of "Decorate any function and now it's a thread on the GPU", but Candle been pretty neat for experimenting with ML on Rust, and easy to move things between CPU and GPU, more of a library than a DSL though: https://github.com/huggingface/candle

Was getting annoyed having to work out which video call platform to use for intro meetings, so I’m currently working on https://simplevc.blazingbanana.com. It’s a private, simple browser video chat. No accounts. You can create a new chat or join via link/code. It also has in browser transcription using Whisper (using https://github.com/huggingface/candle/). Currently fighting getting background blur to work smoothly on Chrome on Android, which is proving surprisingly harder than getting browser transcription working... Also still working on small updates to Whistle, a free offline transcription app available on all platforms (with CUDA builds for Windows/Linux) https://blazingbanana.com/work/whistle

If you need WASM, I think Candle is your current best bet: https://github.com/huggingface/candle

GPT-OSS will run even faster on Blackwell chips because of its hardware support for fp4. If anyone is working on training or inference in Rust, I'm currently working on adding fp8 and fp4 support to cudarc[0] and candle[1]. This is being done so I can support these models in our inference engine for Mixlayer[2]. [0] https://github.com/coreylowman/cudarc/pull/449 [1] https://github.com/huggingface/candle/pull/2989 [2] https://mixlayer.com

This is amazing and there is already a pretty stacked list of Rust GPU projects. This seems to be at an even lower level of abstraction than burn[0] which is lower than candle[1]. I gueds whats left is to add backend(s) that leverage naga and others to the above projects? Feeks like everyone is building on different bases here, though I know the naga work is relatively new. [EDIT] Just to note, burn is the one that focuses most on platform support but it looks like the only backend that uses naga is wgpu... So just use wgpu and it's fine? Yeah basically wgpu/ash (vulkan, metal) or cuda [EDIT2] Another crate closer to this effort: https://github.com/tracel-ai/cubecl [0]: https://github.com/tracel-ai/burn [1]: https://github.com/huggingface/candle/

I heard about burn [1] and candle [2] recently and they sounded interesting. [1] https://crates.io/crates/burn [2] https://github.com/huggingface/candle Interestingly, burn supports candle as a backend.

Surprised this doesn't mention candle: https://github.com/huggingface/candle

You are correct. This project is "on the CPU", so it will not utilize your GPU for computation. If you would like to try out a Rust framework that does support GPUs, Candle https://github.com/huggingface/candle/tree/main may be worth exploring

We used candle[0], which uses cudarc and the metal crate under the hood. That means we run on nvidia hardware in production and can test locally on macbooks with smaller models. I would certainly like to use non nvidia hardware but at this point it's not a priority. The subset of tensor operations needed to run the forward pass of LLMs isn't as large as you'd think though. [0] https://github.com/huggingface/candle

Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use https://github.com/huggingface/candle

Candle already exists[1], and it runs pretty well. Can use both CUDA and Metal backends (or just plain-old CPU). [1] https://github.com/huggingface/candle

I switched from python to rust for my AI stuff. Honestly, I don't care about the things people say rust is used for. I like it because the package manager, testing, and typings being built into the ecosystem by default makes it so easy to build. VS Python where it all can be done, but you need to then maintain all of those separate tools. The overhead of writing Rust is less than the overhead of dealing with the Python ecosystem. And then you have all the benefits of Rust everyone mentions more often... one other thing no one mentions is the feedback loop between a strongly typed language and copilots ability to more accurately generate code. That being said, there is a real shortage of Rust software for Rust only projects. I ended up writing a wrapper for Llama.cpp and open ai API [0] because I needed it and couldn't find anything out th

I deployed a ML solution in Rust back in 2020 (tch-rs instead of tensorflow, though), so this has been possible for a while. Surprisingly I saw some impressive speed gains vs python (and vs JS, tried that one as well), probably because I was running tons on images through it and the percentage of time spent doing inference wasn't very high. Deployment wasn't the easiest, but doable. Instead of jumping on the tensorflow or torch boat, when doing Rust today I would consider using https://github.com/huggingface/candle