stanford_alpaca

Yes this is the point of Stanford Alpaca https://github.com/tatsu-lab/stanford_alpaca They used the larger and more expensive text-davinci-003 model to fine-tune the smaller and cheaper 7B LLaMA model.

I feel like there is an emerging consensus that [Chat]GPT 3.5/4 is not just 1 big model. A large part of the magic in the final product appears to be many intermediate layers of classification that select the appropriate LLM/method to query. The cheaper models (e.g. Ada/Babbage) could be used for this purpose. Think about why offensive ChatGPT prompts are rejected so quickly compared to legitimate asks for code. Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Cross-domain queries could be managed mostly across turns of conversation w/ summarization. Think about the Stanford Alpaca parse analysis diagram [0]. You could have an instruction-following model per initial keyword. All of "Write..." might fit into a much smaller model if isolated. This stuff could be parti

Thanks! - Training code is https://github.com/tatsu-lab/stanford_alpaca - Params were mostly default (as in stanford_alpaca README), except for: per_device_train_batch_size=1, per_device_eval_batch_size=1 - Fine-tuning dataset was based on https://github.com/tloen/alpaca-lora/raw/81eb72f707b0505a03b... with minor improvements; I'm going to publish my version soon - The training itself took about 3 hours on 8x Nvidia A100 80GB

The original Alpaca repo has the training script. The readme has the torchrun command and arguments used for train.py. https://github.com/tatsu-lab/stanford_alpaca/blob/main/train...

Creating prompts to get good results is a skill. - For text prompting, a good starting point is reading the whitepapers in this repo [0]. - For AI art generation, a good starting point are AI prompt libraries to see results of good prompts. Fine-tuning (and creating the examples for fine-tuning) is also an interesting challenge. - The Stanford Alpaca repo [1] made it really easy to fine-tune LLaMa based on your custom dataset. [0] - https://github.com/promptslab/Awesome-Prompt-Engineering [1] - https://github.com/tatsu-lab/stanford_alpaca

On their GitHub repo (https://github.com/tatsu-lab/stanford_alpaca) they've added a notice: "Note: Due to safety concerns raised by the community, we have decided to shut down the Alpaca live demo. Thank you to everyone who provided valuable feedback." So probably this was the usual type of people complaining about the usual type of thing.

Training code: https://github.com/tatsu-lab/stanford_alpaca#fine-tuning

Stanford released the exact training data as well as the training script with all parameters. Boot up a p4.2xlarge (8 A100 GPUs) which costs about $40/hour and let it run for a 2-3 hours and voila. See the Readme in their repo where it mentions the fine-tuning script[0] [0] https://github.com/tatsu-lab/stanford_alpaca

If they don't release the model, recreating it doesn't look too hard. $100 worth of compute time to run the fine-tuning, and the training data they used is here: https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpac... That would have the same licensing problems that they have though: that alpaca_data.json file was created using GPT3. But creating a "clean" training set of 52,000 examples doesn't feel impossible to me for the right group.

https://github.com/tatsu-lab/stanford_alpaca Tada! Literally runs on a raspberry pi (very slowly). GPT models are incredible but the future is somehow even more amazing than that. I suspect this will be the approach for legal / medical uses (if regulation allows).

It's $3.80 for the full 1000 prompt tokens plus 32k token long context. But OpenAI doesn't have a monopoly on long context, RWKV is free and open source and has virtually an unlimited context (as long as you remind it every once and a while). However ChatGPT really cannot be matched at this point perhaps except LLaMa Alpaca 7B EDIT: I accidentially typed Alpine, it's Alpaca. https://github.com/tatsu-lab/stanford_alpaca

Here's a link that opens their training data (52,000 rows) in Datasette Lite: https://lite.datasette.io/?json=https://github.com/tatsu-lab... Means you can run SQL LIKE queries against it to try and get a feel for what's in there. This link includes a query for running searches for terms: https://lite.datasette.io/?json=https://github.com/tatsu-lab...