skypilot

To massively increase the reliability to get GPUs, you can use something like SkyPilot (https://github.com/skypilot-org/skypilot) to fall back across regions, clouds, or GPU choices. E.g., $ sky launch --gpus H100 will fall back across GCP regions, AWS, your clusters, etc. There are options to say try either H100 or H200 or A100 or <insert>. Essentially the way you deal with it is to increase the infra search space.

Skypilot is worth a mention here: https://github.com/skypilot-org/skypilot Open source CLI to deploy multiple gpu vm’s on all major cloud providers, with an option to use spot pricing with 1 cheap vm used as a controller to always make sure you have the most inexpensive deployment available with failover and load balancing. It’s like beating the cloud providers at their own game I wouldn’t be surprised if they banned it.

I recommend reviewing Stanford's dspy library - great examples of few-shot learning that works by generating and tuning prompts for LLMs and even distilling instruction following tasks to smaller models like T5. Second, as others mentioned, using QLoRA for supervised fine-tuning followed by DPO/KTO for preference optimization. This strategy placed Huggingface's Zephyr and IBM's Neural Chat on leaderboards for 7B parameter models. I also recommend reviewing the Unsloth library which has excellent accelerated examples of using these methods, along with the axolotl library. Lastly, skypilot and Modal both have excellent examples that showcase using axolotl to efficiently finetune models on cloud GPUs. [1] https://github.com/stanfordnlp/dspy [2] https://github.com/unslothai/unsloth [3] https://github.com/OpenAccess-AI-Collective/axolotl [4] h

Great to see this effort! Couldn't think of a better duo to experiment with it! >with academic groups to help support research that might help make AI more accessible Jeremy, are you already talking with Sky Computing Lab? I recall they had an interesting project about SkyPilot which seemed helpful https://sky.cs.berkeley.edu/ https://github.com/skypilot-org/skypilot

It sounds like you might be wanting something for GPU batch job management. Some things to check out would be gpu orchestration tools, specifically: Slurm, Run.ai, and Skypilot. Or maybe you're kind of wanting a serverless GPU cloud - check out Runpod, Modal, Baseten, and Replicate. Links: https://slurm.schedmd.com/documentation.html https://www.run.ai/ml-workflow-management https://github.com/skypilot-org/skypilot https://www.runpod.io/serverless-gpu https://modal.com/pricing https://www.baseten.co/pricing/ https://replicate.com/pricing

I have been researching methods and projects built around training and running LLMs locally. I'm interested in what others have been using on this front (including straight up Pytorch/Transformers). Here's what I've gathered so far: Engines/APIs: - vllm: Inference and serving engine for LLMs (none quantized models only?) [1] - ollama: Go project to run, create and share LLMs [2] - llama.cpp: Inference of LLaMA models in C/C++ w/UI (including quantized models) [3] - llama-cpp-python: run OpenAI compatible API bindings for llama.cpp [4] - llm-engine: engine for fine-tuning and serving LLMs [5] - Lamini: hosted? closed source? solution for training LLMs [6] - GPT4All: free, locally run chatbot [7] - SkyPilot: framework for running LLMs, AI, and batch jobs [8] - HuggingFace Transformers: APIs and tools to download and train