KAKarrot_Kream·Sep 28, 2023
Take a look at the QLoRA repo https://github.com/artidoro/qlora/ which has an example finetuning Llama. Made by the authors of the QLoRA paper.
In Mistral 7Bartidoro
qlora
Jupyter Notebook2023–2023
artidoro / qlora
qlora
QLoRA: Efficient Finetuning of Quantized LLMs
7Distinct people
7Mentions
0Explicit recommenders
11.1KGitHub stars
What to know before exploring
What it is
QLoRA: Efficient Finetuning of Quantized LLMs
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by week7 dated mentions
Observed mentions: 2023-05-24 16:59:37 UTC to 2023-09-28 00:09:17 UTC. Weeks start Monday, UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| May 22 | 2023-05-22 00:00:00 UTC | 2023-05-29 00:00:00 UTC | 2 | 1 | 1 |
| May 29 | 2023-05-29 00:00:00 UTC | 2023-06-05 00:00:00 UTC | 1 | 0 | 1 |
| Jun 5 | 2023-06-05 00:00:00 UTC | 2023-06-12 00:00:00 UTC | 0 | 0 | 0 |
| Jun 12 | 2023-06-12 00:00:00 UTC | 2023-06-19 00:00:00 UTC | 0 | 0 | 0 |
| Jun 19 | 2023-06-19 00:00:00 UTC | 2023-06-26 00:00:00 UTC | 0 | 0 | 0 |
| Jun 26 | 2023-06-26 00:00:00 UTC | 2023-07-03 00:00:00 UTC | 1 | 0 | 1 |
| Jul 3 | 2023-07-03 00:00:00 UTC | 2023-07-10 00:00:00 UTC | 0 | 0 | 0 |
| Jul 10 | 2023-07-10 00:00:00 UTC | 2023-07-17 00:00:00 UTC | 0 | 0 | 0 |
| Jul 17 | 2023-07-17 00:00:00 UTC | 2023-07-24 00:00:00 UTC | 0 | 0 | 0 |
| Jul 24 | 2023-07-24 00:00:00 UTC | 2023-07-31 00:00:00 UTC | 1 | 0 | 1 |
| Jul 31 | 2023-07-31 00:00:00 UTC | 2023-08-07 00:00:00 UTC | 0 | 0 | 0 |
| Aug 7 | 2023-08-07 00:00:00 UTC | 2023-08-14 00:00:00 UTC | 0 | 0 | 0 |
| Aug 14 | 2023-08-14 00:00:00 UTC | 2023-08-21 00:00:00 UTC | 1 | 1 | 0 |
| Aug 21 | 2023-08-21 00:00:00 UTC | 2023-08-28 00:00:00 UTC | 0 | 0 | 0 |
| Aug 28 | 2023-08-28 00:00:00 UTC | 2023-09-04 00:00:00 UTC | 0 | 0 | 0 |
| Sep 4 | 2023-09-04 00:00:00 UTC | 2023-09-11 00:00:00 UTC | 0 | 0 | 0 |
| Sep 11 | 2023-09-11 00:00:00 UTC | 2023-09-18 00:00:00 UTC | 0 | 0 | 0 |
| Sep 18 | 2023-09-18 00:00:00 UTC | 2023-09-25 00:00:00 UTC | 0 | 0 | 0 |
| Sep 25 | 2023-09-25 00:00:00 UTC | 2023-10-02 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
7 sourcesAsk HN: Are you training and running custom LLMs and how are you doing it?
I have been researching methods and projects built around training and running LLMs locally. I'm interested in what others have been using on this front (including straight up Pytorch/Transformers). Here's what I've gathered so far:
Engines/APIs:
- vllm: Inference and serving engine for LLMs (none quantized models only?) [1]
- ollama: Go project to run, create and share LLMs [2]
- llama.cpp: Inference of LLaMA models in C/C++ w/UI (including quantized models) [3]
- llama-cpp-python: run OpenAI compatible API bindings for llama.cpp [4]
- llm-engine: engine for fine-tuning and serving LLMs [5]
- Lamini: hosted? closed source? solution for training LLMs [6]
- GPT4All: free, locally run chatbot [7]
- SkyPilot: framework for running LLMs, AI, and batch jobs [8]
- HuggingFace Transformers: APIs and tools to download and train
Read the rest
models (via Pytorch) [9]
- RAGStack: Deploy a private ChatGPT alternative hosted within your VPC [14]
UI/Interface:
- Simon's `llm` tool [15]
Quantization Bits:
- AutoGPTQ: GPTQ algorithm based quantization package [10]
- QLoRA: finetuning of quantized LLMs [11]
- bitsandbytes: CUDA functions for PyTorch [12]
- SkyPilot QLoRA [13]
Video Guides:
- https://www.youtube.com/watch?v=eeM6V5aPjhk
- https://www.youtube.com/watch?v=TYgtG2Th6fI (jawerty's example)
Reference:
- [1] https://github.com/vllm-project/vllm
- [2] https://github.com/jmorganca/ollama
- [3] https://github.com/ggerganov/llama.cpp
- [4] https://github.com/abetlen/llama-cpp-python
- [5] https://github.com/scaleapi/llm-engine
- [6] https://www.lamini.ai/
- [7] https://github.com/nomic-ai/gpt4all
- [8] https://github.com/skypilot-org/skypilot
- [9] https://github.com/huggingface/transformers/releases
- [10] https://github.com/PanQiWei/AutoGPTQ
- [11] https://github.com/artidoro/qlora
- [12] https://github.com/TimDettmers/bitsandbytes
- [13] https://github.com/artidoro/qlora/pull/132
- [14] https://github.com/psychic-api/rag-stack
- [15] https://simonwillison.net/2023/Aug/1/llama-2-mac/
This is an ad. You'd be best served avoiding additional dependencies. At this point, you don't want to be trading off simplicity for ease. Even transformers + huggingface feels like too much bloat.
You can use this
https://github.com/PygmalionAI/training-code
Or, you can use this; for QLoRA
https://github.com/artidoro/qlora
The tools and mechanisms to get a model to do what you want is ever so changing, ever so quickly. Build and understand a notebook yourself, and reduce dependencies. You will need to switch them.
In Tuning and Testing Llama 2, Flan-T5, and GPT-J with LoRA, Sematic, and GradioI used: https://github.com/artidoro/qlora but there are quite a few others that likely work better. It was literally my first attempt at doing anything like this, and took the better part of an evening to work through CUDA/Python issues to get it training, and ~20 hours of training.
In Databricks Strikes $1.3B Deal for Generative AI Startup MosaicMLHow much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora. But as other comments mention, it's still noticeably worse in my experience.
In Ask HN: What's the best self hosted/local alternative to GPT-4?Currently SOTA for specialization of LLMs is QLoRA: https://github.com/artidoro/qlora
In The False Promise of Imitating Proprietary LLMsQLoRA 4-bit finetuning of LLMs
7 points · Open discussion