peft

PEFT is more niche as it's a specific library (https://github.com/huggingface/peft) that creates LoRAs, but for a machine learning engineer working with LLM models outside of just prompting an API, it's within scope to know why/how a LoRA should be used.

There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft)

Is it? Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence. E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68... Take this line: pip3 install torch torchvision torchaudio Which version of torch is this? The latest. FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04 Which version of CUDA is this? An incompatible one, apparently. Game over. Check out "requirements.txt": accelerate==0.18.0 colorama datasets flexgen==0.1.7 gradio==3.25.0 markdown numpy pandas Pillow>=9.5.0 pyyaml requests rwkv==0.7.3 safetensors==0.3.0 sentencepiece tqdm Wow. Less than half of those have any version specified. The rest? "Meh, I don't care, whatever." Then this beauty: git+https://gith

I haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers. Check out https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA (https://arxiv.org/pdf/2106.09685.pdf) That should get you started.

Adding this to the huggingface peft library would be amazing. That's the main library that people using LoRA are currently using. https://github.com/huggingface/peft/issues/289

TIL learned about LoRA via https://news.ycombinator.com/item?id=35287740 See also: Huggingface PEFT: https://github.com/huggingface/peft

> With no real knowledge of LLM and only recently started to understand what LLM terms mean, such as 'model, inference, LLM model, intruction set, fine tuning' whatelse do you think is required to make a took like yours? This was mee a few weeks ago. I got interested in all this when FlexGen (https://github.com/FMInference/FlexGen) was announced, which allowed to run inference using OPT model on consumer hardware. I'm an avid user of Stable Diffusion, and I wanted to see if I can have an SD equivalent of ChatGPT. Not understanding the details of hyperparameters or terminology, I basically asked ChatGPT to explain to me what these things are: Explain to someone who is a software engineer with limited knowledge of ML terms or linear algebra, what is "feed forward" and "self-attention" in the context of ML and large language models. Provi