llm

> Tensorflow is a C++ framework that has Python bindings and a Python library, but when the models are served they are running on C++ Sure, and it's only a simple 20 step process that involves building Tensorflow from source. Yeay! https://medium.com/@hamedmp/exporting-trained-tensorflow-mod... Let me see what the process for compiling a LLM written in Rust is.... https://github.com/rustformers/llm cargo install llm-cli Oh look, it doesn't make me immediately want to give up.

G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for running inference on large language models (of the kind that power ChatGPT and more); we provide a CLI and a Rust crate for running inference on these models, all entirely open-source. The crate can be embedded in your own projects, allowing you to ea

Hi! I'm a maintainer of https://github.com/rustformers/llama-rs. We're planning to expand our model support soon.

I'm curious if someone will have to port these enhancements elsewhere, ie: https://github.com/rustformers/llama-rs

I have not, but I want to in near future because I'm really curious myself too. I've been following Rust community that now has llama.cpp port and also my OpenCL thing and one discussion item has been to run a verification and common benchmark for the implementations. https://github.com/setzer22/llama-rs/issues/4 I've mostly heard that, at least for the larger models, quantization has barely any noticeable effect. Would be nice to witness it myself.

I've counted three different Rust LLaMA implementations on r/rust subreddit this week: https://github.com/Noeda/rllama/ (pure Rust+OpenCL) https://github.com/setzer22/llama-rs/ (ggml based) https://github.com/philpax/ggllama (also ggml based) There's also a discussion on GitHub issue on setzer's repo to collaborate a bit on these separate efforts: https://github.com/setzer22/llama-rs/issues/4

Great job porting the C++ code! Seems like the reasoning was to provide the code as a library to embed in a HTTP Server, cannot wait to see that happen and try it out. Looking at how the inference runs, this shouldn't be a big problem, right? https://github.com/setzer22/llama-rs/blob/main/llama-rs/src/...