llm-foundry

Related comment from gwern: https://news.ycombinator.com/item?id=38438859. Can't find the docs now - I think they were the old GPT 3 ones - but they suggested a low value somewhere around 0.01 and 0.1. Also - why qlora rather than a full finetune? Using LambdaLabs, it'd cost roughly the same as your quote. Cheaper I think if you're willing to gamble with fp8: https://github.com/mosaicml/llm-foundry/tree/main/scripts/tr.... And fewer hyperparameters to tune as well

Most AI runners just ignore licensing and (illegally?) run LLaMA finetunes. There are tons of good 33B/65B instruct models you can run on any decent PC. But if you want to avoid the non commercial LLaMA license, you have 3 good options for a base model. - OpenLlama 13B - MPT 30B - Falcon 40B Of these, Falcon 40B is very difficult to run (slow in 4 bit, basically requires a professional GPU, no good cpu offloading yet). OpenLLaMA 13B only supports a context size of 2048 as of today... But that could change soon. So you probably want MPT instruct 30B, specifically this one: https://huggingface.co/TheBloke/mpt-30B-instruct-GGML As the page says, you can try it out on a decent PC of your own with the OpenCL build of KoboldCPP. Change it to "instruct" mode, use the template on the page, offload as many layers as you can to your PC's dGPU, and

The repo for training and finetuning this model is open source here: https://github.com/mosaicml/llm-foundry