axolotl

I usually use https://github.com/axolotl-ai-cloud/axolotl on Lambda/Together for working with these types of models. Curious what others are using? What is the quickest way to get started? They mention Pre-training and Post-training but sadly didnt provide any reference starter scripts.

Liger-Kernel support has already been merged to axolotl [0] along with an example config that makes use of it [1], if anyone would like to quickly try it out. [0] https://github.com/axolotl-ai-cloud/axolotl/pull/1861 [1] https://github.com/axolotl-ai-cloud/axolotl/blob/main/exampl...

I recommend reviewing Stanford's dspy library - great examples of few-shot learning that works by generating and tuning prompts for LLMs and even distilling instruction following tasks to smaller models like T5. Second, as others mentioned, using QLoRA for supervised fine-tuning followed by DPO/KTO for preference optimization. This strategy placed Huggingface's Zephyr and IBM's Neural Chat on leaderboards for 7B parameter models. I also recommend reviewing the Unsloth library which has excellent accelerated examples of using these methods, along with the axolotl library. Lastly, skypilot and Modal both have excellent examples that showcase using axolotl to efficiently finetune models on cloud GPUs. [1] https://github.com/stanfordnlp/dspy [2] https://github.com/unslothai/unsloth [3] https://github.com/OpenAccess-AI-Collective/axolotl [4] h

The approach I see used is axolotl with QLoRA using cloud GPUs which can be quite cheap. https://github.com/OpenAccess-AI-Collective/axolotl Someone from one of the cloud GPU vendors wrote a guide: https://brev.dev/blog/fine-tuning-mistral