candle

In my case, I greatly admire what folks at Huggingface have built in the candle[0] project. The project itself is amazing. I haven't yet delved deeper into the source code, but the code in their examples is clear and easy to understand [0] https://github.com/huggingface/candle

We like Rust for its build tooling and safe concurrency primitives. Thus we are eyeballing with candle [1]. OTOH, writing platform specific backends is a huge undertaking. Do you think backends such as MPS may become a shared effort? [1] https://github.com/huggingface/candle [2] https://github.com/huggingface/candle/issues/313

How did you select those two as your options? I'm just a hobbyist that enjoys programming, and I eventually wanted to expand beyond python. I looked at Haskell and read Learn You a Haskell and did some Exercism exercises but never got anywhere close to being able to use it for real projects. Have been trying to learn about Lisp lately and feel like I've come to a similar dead end. On the other hand, both Go and Rust have felt fulfilling and practical, with static typing and solid tooling, cross compilations, static binaries, and dependency management that is just a huge breath of fresh air coming from python. The ML / data science scene is nowhere near as developed as in Python, and I still lean on jupyter/polars/PyTorch here, but I think the candle project[0] seems very interesting. Compiling whisper down to a single CUDA-leveraging bina

Powered by Candle Rust compiled to Wasm: https://github.com/huggingface/candle

https://github.com/huggingface/candle/blob/main/candle-kerne... That’s the C++ variant of cuda, not Rust. They’re just binding it as one might in Python. At which point, what’s the value add?

They already support quantization out of the box! https://github.com/huggingface/candle/pull/314/files

Exciting tech, looking forward to see it running on wgpu! There is an issue about WebGPU although I don’t notice references to the native version of it. https://github.com/huggingface/candle/issues/344

I sense that most folks will end up using this via a Python wrapper, if at all, much like how HF has Transformers written in Rust but with a Python wrapper. This library is targeting production, a much smaller subset of engineers. See the 'why candle' section: https://github.com/huggingface/candle#why-candle. So you could use PyTorch to build models and Candle to serve them.

Very nice! I wanted to do something like this but then I would miss on proper CUDA acceleration and lose performance compared to using torchlib. I wrote a forgettable llama implementation for https://github.com/LaurentMazare/tch-rs (pytorch's torchlib rust binding). Still not ideal but at least you get the same GPU performance you would get on pytorch. ...And then I spotted Candle, a new ML framework by the same author: https://github.com/huggingface/candle It's all in Rust, self contained, a huge undertaking, but it looks very promising. They already have a llama2 example!