LoRA

I wonder how far you can push RAG and it's comparison to methods like LoRA https://github.com/microsoft/LoRA It seems like you can only get so far with a sophisticated prompt engineering framework like RAG. but I dunno, I haven't tested them against each other.

Curious if we'll see a Civitai-style LoRA[1] marketplace for text-to-speech models. 1 = https://github.com/microsoft/LoRA

Yes, your understanding is correct. However, instead of adding a head on top of the network, most fine-tuning is currently done with LoRA (https://github.com/microsoft/LoRA). This introduces low-rank matrices between different layers of your models, those are then trained using your training data while the rest of the models' weights are frozen.

I just read the paper about LORA. The main idea is that you write the weights of each neural network as W = W0 + B A Where W0 is the trained model’s weights, which are kept fixed, and A and B are matrices but with a much much lower rank than the originals (say r = 4). It has been shown (as mentioned in the lora paper that training for specific tasks results in low rank corrections, so this is what it is all about. I think that doing LoRa can be done locally. [1] https://github.com/microsoft/LoRA

Microsoft also reused LoRA in the LLM context recently https://github.com/microsoft/LoRA sigh

There's a difference between buzzwords and jargon. Buzzwords can start out as jargon, but have their technical meaning stripped by users who are just trying to sound persuasive. Examples include words like synergy, vertical, dynamic, cyber strategy, and NFT. That's not what's happening in the parent comment. They're talking about projects like https://github.com/ZrrSkywalker/LLaMA-Adapter https://github.com/microsoft/LoRA https://github.com/tloen/alpaca-lora and specifically the paper: https://arxiv.org/pdf/2106.09685.pdf Lora is just a way to re-train a network for less effort. Before we had to fiddle with all the weights, but with Lora we're only touching 1 in every 10,000 weights. The parent comment says GPT4all doesn't give us a way to train the full size Llama model using the new lora technique. We'll have to build that ourselves. Bu

LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney. With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter... LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/m