alpaca-lora

There are tons of them for other models, often Alpaca or LLaMA. This is a popular tool for finetuning Alpaca: https://github.com/tloen/alpaca-lora

Not just inference. AFAIK, you are able to fine-tune the models with custom data[1], which does not seem to require anything but a GPU with enough VRAM to fit the model in question. I'm looking to get my hands on an RTX 4090 to ingest all of the repair manuals of a certain company and have a chatbot capable of guiding repairs, or at least try to do so. So far doing inference only as well. [1] https://github.com/tloen/alpaca-lora

There's a difference between buzzwords and jargon. Buzzwords can start out as jargon, but have their technical meaning stripped by users who are just trying to sound persuasive. Examples include words like synergy, vertical, dynamic, cyber strategy, and NFT. That's not what's happening in the parent comment. They're talking about projects like https://github.com/ZrrSkywalker/LLaMA-Adapter https://github.com/microsoft/LoRA https://github.com/tloen/alpaca-lora and specifically the paper: https://arxiv.org/pdf/2106.09685.pdf Lora is just a way to re-train a network for less effort. Before we had to fiddle with all the weights, but with Lora we're only touching 1 in every 10,000 weights. The parent comment says GPT4all doesn't give us a way to train the full size Llama model using the new lora technique. We'll have to build that ourselves. Bu

There's a neat trick when you encounter jargon. 1. Identify the jargon terms you don't understand 2. Lookup papers that introduce the jargon terms 3. Skim-read the paper to get the gist of the jargon If you don't want to do this, then you don't have to feel uneducated. You can simply choose to feel like your time is more important than skimming a dozen AI papers a week. But for example, here's what I did to understand the parent comment: 1. I had no idea what lora is or how it relates to alpaca. 2. I looked up https://github.com/tloen/alpaca-lora 3. I read the abstract of the Lora paper: https://arxiv.org/pdf/2106.09685.pdf https://github.com/tloen/alpaca-lora 4. Now I know that Lora is just a way of using low rank matrices to reduce finetuning difficulty by a factor of like 10,000 or something ridiculous 5. Since I don't actually care ab

For those wondering why this is interesting: This technique is being used to reproduce[0] the Alpaca results from Stanford[1] with a few hours of training on consumer-grade hardware. I believe there will soon be a cottage industry of providing application-specific fine-tuned models like this, that can run in e.g. AWS very inexpensively. The barrier today seems to be that the base model (here, Meta's LLaMA) is encumbered and can't be used commercially. Someone will soon, I'm confident, release e.g. an MIT-licensed equivalent and we'll all be off to the races. [0] https://github.com/tloen/alpaca-lora [1] https://crfm.stanford.edu/2023/03/13/alpaca.html

Thanks! - Training code is https://github.com/tatsu-lab/stanford_alpaca - Params were mostly default (as in stanford_alpaca README), except for: per_device_train_batch_size=1, per_device_eval_batch_size=1 - Fine-tuning dataset was based on https://github.com/tloen/alpaca-lora/raw/81eb72f707b0505a03b... with minor improvements; I'm going to publish my version soon - The training itself took about 3 hours on 8x Nvidia A100 80GB

did you use the cleaned and improved alpaca dataset from https://github.com/tloen/alpaca-lora/issues/28 ?

I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!

Apparently can fine tune and run Alpaca-lora 7B using a single RTX 4090: https://github.com/tloen/alpaca-lora A few days ago before it was taken offline I did some tests I did with Alpaca-stanford. If I can run a GPT-3 level LLM in my machine, I won't even bother looking at GPT-4. Yes GPT-4 is impressive, but GPT-3 and Alpaca are already good enough IMO.

Use this Alpaca replication instead https://github.com/tloen/alpaca-lora

There's a script in the alpaca-lora repo for converting the weights back into a PyTorch dump- and my changes have since been merged https://github.com/tloen/alpaca-lora/pull/19