rllama

Location: San Francisco Remote: No preference, as long as I don't have to move far from Bay Area Willing to relocate: No Technologies: C, Rust, Golang, Haskell, Lisp, Python, Lua, OpenGL, SQLite3, JavaScript, PostgreSQL, AWS EC2, S3, ECS, Batch. Resume: https://www.linkedin.com/in/mikjuola Email: mikko.juola@iki.fi --- I've been working at the Bay Area since 2015, most recently at Pinterest. At work, I've done big data pipelines, designed some batch job systems, computing metrics, handling billing APIs, lots of Python, Go and Java and working with AWS, i.e. backend and data engineer stuff. I've been coding since I was less than 10 years old so I am quite strong in churning out code. But I'm trying to look for work that's more in line with what I do on my free time: Challenging low-level C or Rust programming, machine learning implementati

I work on an independent LLM implementation here: https://github.com/Noeda/rllama/ I only got it working at all yesterday and there's no nice UX at all. Not sure I recommend trying to use this as llama.cpp will probably have this in no time with a much better user experience, although I am also trying to make it more usable. If you follow the instructions on Vicuna page over how to apply the deltas, and you can compile the project, then you could run: cargo run --release --features opencl -- --model-path /models/vicuna13b --param-path /models/vicuna13b/config.json --tokenizer-path /models/vicuna13b/tokenizer.model --prompt-file prompt --top-p 1.0 --top-k 20 --repetition-penalty 1 --temperature 0.9 --max-seq-len 2048 --f16 --percentage-to-gpu 0.9 Where /models/vicuna13b is the HuggingFace-compatible model. This will put 90% of weights on G

No, my project is called rllama. No relation to GGML. https://github.com/Noeda/rllama

I ran it on a 128 RAM machine with a Ryzen 5950X. It's not fast, 4 seconds per token. But it's just about fits without swapping. https://github.com/Noeda/rllama/

I've counted three different Rust LLaMA implementations on r/rust subreddit this week: https://github.com/Noeda/rllama/ (pure Rust+OpenCL) https://github.com/setzer22/llama-rs/ (ggml based) https://github.com/philpax/ggllama (also ggml based) There's also a discussion on GitHub issue on setzer's repo to collaborate a bit on these separate efforts: https://github.com/setzer22/llama-rs/issues/4