Soup

There are some samples of training data in this folder - https://github.com/MakazhanAlpamys/Soup/tree/main/examples/d... They're all very short though. Anyone got a good rule for how much data of this nature is needed to successfully fine-tune a model of this size?

Author here. The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model. Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can c