rust-bert

What are the respective ecosystems of these languages like for ML/NLP stuff? From what I can tell, the one with the most mature/active ecosystem is Rust. There is both Candle, a PyTorch equivalent, and a lib that (per my understanding) seeks to be 1:1 with the HuggingFace Python Transformers library (https://github.com/guillaume-be/rust-bert). Haskell has HaskTorch, and OCaml has some OCaml thing that for some reason invokes PyTorch under the hood instead of just interfacing with the C++ deps directly. But it looks like there isn't support for doing things like replicating the Transformers library, and similarly maybe not a lot of support for issues and needs I haven't even run into yet (?)

I'd like to use this transformer model in rust (because it's on the backend, because I can use data munging and it will be faster, and for other reasons). It looks like a good model! But, it doesn't compile on Apple Silicon for wierd linking issues that aren't apparent - https://github.com/guillaume-be/rust-bert/issues/338. I've spent a large part of today and yesterday attempting to find out why. The only other library that I've found for doing this kind of thing programmatically (particularly sentiment analysis) is this (https://github.com/JohnSnowLabs/spark-nlp). Some of the models look a little older, which is OK, but it does mean that I'd have to do this in another language. Does anyone know of any sentiment analysis software that can be tuned (other than VADER - I'm looking for more along the lines of a transformer model) - like BER

Not at all, I've got 2 models running free-of-charge on Oracle Cloud right now. Just playing with it locally is completely free, the hardest part is getting your Python environment to work. I wrote mine in Rust and even that isn't super complex. It mostly boils down to defining your settings and feeding it your prompt: https://github.com/guillaume-be/rust-bert/blob/master/exampl... Training is more difficult, but this will be plenty for your purposes I think.

Give this a look: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert/blob/master/exampl... If you have Pytorch configured correctly, this should "just work" for a lot of the smaller models. It won't be a 1:1 ChatGPT replacement, but you can build some pretty cool stuff with it. > it's basically Python or bust in this space More or less, but that doesn't have to be a bad thing. If you're on Apple Silicon, you have plenty of performance headroom to deploy Python code for this. I've gotten this library to work on systems with as little as 2gb of memory, so outside of ultra-low-end use cases, you should be fine.

I suspect something similar is possible with ChatGPT. Using the GPT-neo-125m model I've been able to get some really convincing (if lackluster) answers on 4 core ARM hardware and less than 2gb of memory. With enough sampling, you can get legible paragraph-length responses out in less than 10 seconds; that's pretty good for an offline program in my book. I'm using rust-bert to serve it over a Discord bot, similar to one of their examples[0]. It's running on Oracle VCPUs right now, but with dedi hardware and ML acceleration I bet it would scream! [0] https://github.com/guillaume-be/rust-bert/blob/master/exampl...

Maybe a little played-out, but I've been having a blast with the rust-bert library this weekend: https://github.com/guillaume-be/rust-bert With a little fanagling, you can get the GPT-Neo-1.3b model running on those free Oracle ARM VMs. I'm impressed, especially with the performance of the smallest model that uses less than a gig of memory.

Not as far as I'm aware, but it's probably possible on a smaller, hobbyist scale. I've been playing with a few GPT libraries this week (namely rust-bert[0]) and I've been really impressed with local generation results on my crappy 2 core netbook. I can get 2 sentences to generate in ~5 seconds, which is pretty good in my book. Armed with a Pi-style SBC and your AI library of choice, I bet you could get pretty far implementing some toy-like behavior. Bonus points if you use Whisper for speech-to-text, and double brownie points if you can get an AI voice to read the generation back. [0] https://github.com/guillaume-be/rust-bert/

As a starting point, maybe take a look at the 'tch' crate [1] and the 'rust-bert' crate [2] built on top of it? [1]: https://github.com/LaurentMazare/tch-rs -> https://crates.io/crates/tch [2]: https://github.com/guillaume-be/rust-bert -> https://crates.io/crates/rust-bert