lit-llama

Compete with Llama.cpp? Like transformers llama [0], exllama [1] (really fast), or litllama [2] ? exllama is really memory efficient and really fast [0] https://huggingface.co/docs/transformers/main/model_doc/llam... [1] https://github.com/turboderp/exllama [2] https://github.com/Lightning-AI/lit-llama EDIT: Or do you mean cuda? Because yeah, it's such a shame AMD's Rocm is so bad even geohot gave up. it's examples don't even run without crashing. https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec...

So is https://github.com/Lightning-AI/lit-llama

You can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model What OP did is more intense, but Lora/Adapter still can give excellent results. Pure AI cards for mere mortals aren't really a thing yet.

This is fabulous. Just want to add that there are efforts to impove training speed, like this: https://github.com/Lightning-AI/lit-llama/issues/62 So the practical cost/dataset size for language finetunes is bound to get better rapidly. EDIT: And there is also this for JAX finetuning. https://github.com/young-geng/EasyLM/blob/main/docs/llama.md

We believe that AI should be fully open source and part of the collective knowledge. The original LLaMA code is GPL licensed which means any project using it must also be released under GPL. This "taints" any other code and prevents meaningful academic and commercial use. Lit-LLaMA solves that for good.