MNmnoorfawi·Jul 21, 2024Submission
Show HN: Fine-tune LLMs without catastrophic forgetting with CURLoRA approach
3 points · Open discussionMNoorFawi
curlora
Jupyter Notebook2024–2024
MNoorFawi / curlora
curlora
The code repository for the CURLoRA research paper. Stable LLM continual fine-tuning and catastrophic forgetting mitigation.
2Distinct people
3Mentions
0Explicit recommenders
54GitHub stars
What to know before exploring
What it is
The code repository for the CURLoRA research paper. Stable LLM continual fine-tuning and catastrophic forgetting mitigation.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by 6-hour interval3 dated mentions
Observed mentions: 2024-07-16 08:49:19 UTC to 2024-07-21 14:13:43 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Jul 16, 06:00 | 2024-07-16 06:00:00 UTC | 2024-07-16 12:00:00 UTC | 1 | 0 | 1 |
| Jul 16, 12:00 | 2024-07-16 12:00:00 UTC | 2024-07-16 18:00:00 UTC | 0 | 0 | 0 |
| Jul 16, 18:00 | 2024-07-16 18:00:00 UTC | 2024-07-17 00:00:00 UTC | 0 | 0 | 0 |
| Jul 17, 00:00 | 2024-07-17 00:00:00 UTC | 2024-07-17 06:00:00 UTC | 0 | 0 | 0 |
| Jul 17, 06:00 | 2024-07-17 06:00:00 UTC | 2024-07-17 12:00:00 UTC | 0 | 0 | 0 |
| Jul 17, 12:00 | 2024-07-17 12:00:00 UTC | 2024-07-17 18:00:00 UTC | 0 | 0 | 0 |
| Jul 17, 18:00 | 2024-07-17 18:00:00 UTC | 2024-07-18 00:00:00 UTC | 0 | 0 | 0 |
| Jul 18, 00:00 | 2024-07-18 00:00:00 UTC | 2024-07-18 06:00:00 UTC | 0 | 0 | 0 |
| Jul 18, 06:00 | 2024-07-18 06:00:00 UTC | 2024-07-18 12:00:00 UTC | 1 | 1 | 0 |
| Jul 18, 12:00 | 2024-07-18 12:00:00 UTC | 2024-07-18 18:00:00 UTC | 0 | 0 | 0 |
| Jul 18, 18:00 | 2024-07-18 18:00:00 UTC | 2024-07-19 00:00:00 UTC | 0 | 0 | 0 |
| Jul 19, 00:00 | 2024-07-19 00:00:00 UTC | 2024-07-19 06:00:00 UTC | 0 | 0 | 0 |
| Jul 19, 06:00 | 2024-07-19 06:00:00 UTC | 2024-07-19 12:00:00 UTC | 0 | 0 | 0 |
| Jul 19, 12:00 | 2024-07-19 12:00:00 UTC | 2024-07-19 18:00:00 UTC | 0 | 0 | 0 |
| Jul 19, 18:00 | 2024-07-19 18:00:00 UTC | 2024-07-20 00:00:00 UTC | 0 | 0 | 0 |
| Jul 20, 00:00 | 2024-07-20 00:00:00 UTC | 2024-07-20 06:00:00 UTC | 0 | 0 | 0 |
| Jul 20, 06:00 | 2024-07-20 06:00:00 UTC | 2024-07-20 12:00:00 UTC | 0 | 0 | 0 |
| Jul 20, 12:00 | 2024-07-20 12:00:00 UTC | 2024-07-20 18:00:00 UTC | 0 | 0 | 0 |
| Jul 20, 18:00 | 2024-07-20 18:00:00 UTC | 2024-07-21 00:00:00 UTC | 0 | 0 | 0 |
| Jul 21, 00:00 | 2024-07-21 00:00:00 UTC | 2024-07-21 06:00:00 UTC | 0 | 0 | 0 |
| Jul 21, 06:00 | 2024-07-21 06:00:00 UTC | 2024-07-21 12:00:00 UTC | 0 | 0 | 0 |
| Jul 21, 12:00 | 2024-07-21 12:00:00 UTC | 2024-07-21 18:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
3 sourcesContinually fine-tune large language models without catastrophic forgetting
1 points · Open discussionThanks a lot for your comment. I appreciate it. I believe it involves two points:
1. Choosing C and R:
Normally C and R are chosen from the higher norm rows and columns to be able to reconstruct W or an approximation of it from C@U@R. However you stated it correctly here "but C and R are chosen with respect to the inverse of their contribution towards W, with lower-norm rows and columns more likely to be chosen. In some sense, the CUR of this fine-tuning is as far as possible from the full weight matrix W while still being drawn from it."
So the whole idea is to mitigate catastrophic forgetting by constraining changes to less critical parts of the original weights giving the original matrix some control over the fine-tuned matrices via C and R which are drawn from W not initiated randomly.
2. You are right about the results in MRPC datase
Read the rest
t. This is due to a couple of reasons:
a. The learning rate I chose was relatively slow 2.5e-4
b. not enough text preprocessing was done to increase accuracy as the core idea was to measure catastrophic forgetting more than accuracy.
I actually recently tried different learning rates, epochs, ranks, alpha and different tasks (planning to conduct them in the preprint later) and got different numbers but with the same conclusion; CURLoRA-n was always better both in mitigating catastrophic forgetting with much less trainable parameters while having a good and sometimes better accuracy from LoRA-n.
Here is the paper repo: https://github.com/mnoorfawi/curlora
It has the implementation code ready for both LoRA and CURLoRA.
if you would like, feel free to give it a try on a specific task of yours with different parameters.
Thanks,