MemoRAG

in the ML weeds for too many years at this point. The paper details the actual process, but the TL;DR is that the memory module they use, basically a draft model, does go through a pretraining phase using the redpajama dataset, and then an SFT phase with a different objective. This all happens before and irrespective of the inference-time task (i.e. asking questions about a given text). Also, as has been pointed out in other comments, the draft model could really be any model that supports long context and has decent retrieval performance. So the actual training phases here may be non-essential depending on your infra/cost constraints.

t at random from the book given their surrounding context, and rewarding/penalizing the neural network based on how well it did. The comment mentions GPU memory in the Colab Notebook merely because this process is expensive -- "fine tuning" is the same thing as "training", just with a nearly-complete starting point. Thus the call to `AutoModelForCausalLM.from_pretrained()`. To answer your question explicitly: the fine-tuning step creates a modified version of the base model as an "offline" step, so the memory requirements during inference (aka "online" operation) are unaffected. Both in terms of storage and in terms of GPU VRAM. I'm not the dev tho so obv apologies if I'm off base! I would passionately argue that that step is more of a small addition to the overall pipeline than a core necessity, though. Fine-tuning is really good for teaching a model to recreate style, tone, structure, and other linguistic details, but it's not a very feasible way to teach it facts. That's what "RAG" is for: making up for this deficiency in fine-tuning. In other words, this repo is basically like that post from a few weeks back that was advocating for "modular monoliths" that employ both strategies (monolith vs. microservices) in a deeply collaborative way. And my reaction is the same: I'm not convinced the details of this meshing will be very revolutionary, but the idea itself is deceptively clever!