unsloth

Ollama is not recommended [0], use llama.cpp or more specifically Unsloth Studio which wraps llama.cpp and which has an API mode you can use to hook into Hermes or another agent. Unsloth make both the Studio and the quants which fix various issues with many models [1] as well as implementing new features like MTP and QAT support much sooner than other teams. In general you should read r/LocalLLaMa as it has a lot of updates regarding local models as the field moves fast. [0] https://sleepingrobots.com/dreams/stop-using-ollama/ [1] https://github.com/unslothai/unsloth/discussions/4921

From the README at https://github.com/unslothai/unsloth: "Unsloth uses a dual-licensing model of Apache 2.0 and AGPL-3.0. The core Unsloth package remains licensed under Apache 2.0, while certain optional components, such as the Unsloth Studio UI are licensed under AGPL-3.0."

Uv helps you up though. Use a pyproject.toml and uv sync. Everything will be put into the venv only, nothing spread across the whole system. The pyproject.toml can even handles build env for you, so you no longer need a setup.sh that installs 10 tool in specific order with specific flag to produce working environment. A single uv sync, and the job is done. Plus the result is reproducible, so if this time uv sync work, then it also work next time. Highly recommend if you are still on pip. Note: Take a example that I used to install unsloth with rocm setup that based on unreleased git version dependencies and graphic card specific build flag, all of them can be handled with one command 'uv sync'. This will require a big pile of shell script if doing another way. https://github.com/unslothai/unsloth/issues/4280#issuecommen...

Hey HN! We're excited to release Unsloth Studio - a culmination of many things we wanted to provide to the community - it includes: 1. A Chat UI which has auto healing tool calling, Python & bash code execution, web search, image, docs input + more! 2. Finetuning of audio, vision, LLMs with an Auto AI Assist data prep 3. Supports GGUFs, Mac, Windows, Linux + audio gen 4. Has SVG rendering in browser, exporting to GGUF 5. gpt-oss harmony rendering, all inference parameters are pre-set and recommended 6. Data designer + synthetic data generation 7. Fast parallel data prep + embedding finetuning 8. And much much more! To get it, run: pip install unsloth unsloth studio setup unsloth studio -H 0.0.0.0 -p 8888 Suggestions are welcome, and we're excited for contributions and for you all to try it out! Appreciate you all!

> which are never, ever written in Python nah never ever ever ever ever ... except https://github.com/FlagOpen/FlagGems https://github.com/linkedin/Liger-Kernel https://github.com/meta-pytorch/applied-ai https://github.com/gpu-mode/triton-index https://github.com/AlibabaPAI/FLASHNN https://github.com/unslothai/unsloth the number of people commenting on this stuff that don't know what they're actually talking about grows by leaps and bounds every day...

https://github.com/unslothai/unsloth I'm not sure if it contains exactly what you're looking for, but it includes several resources and notebooks related to fine-tuning LLMs (including LoRA) that I found useful.

Oh for finetuning - we do have some code for MoE finetuning for Qwen at https://github.com/unslothai/unsloth, but we haven't yet announced it yet!

If you guys need help on GGUFs + Unsloth dynamic quants + finetuning support via Unsloth https://github.com/unslothai/unsloth on day 0 / 1, more than happy to help :)

Hi, Yes you can. The community creates quantized variants of these that can run on consumer GPUs. A 4-bit quantization of LLAMA 70b works pretty well on Macbook pros, the neural engine with unified CPU memory is quite solid for these. GPUs is a bit tougher because consumer GPU RAM is still kinda small. You can also fine-tune them. There are lot of frameworks like unsloth that make this easier. https://github.com/unslothai/unsloth . Fine-tuning can be pretty tricky to get right, you need to be aware of things like learning rates, but there are good resources on the internet where a lot of hobbyists have gotten things working. You do not need a PhD in ML to accomplish this. You will, however, need data that you can represent textually. Source: Director of Engineering for model serving at Databricks.

Unsloth (YC S24) | Machine Learning, Infra & Software Engineers | Full-time/Intern | SF (Pref)/Remote | $100K-$250K pa + equity | https://unsloth.ai/ Come join us! We have 2.5 million monthly model downloads, 17.6K Github stars & collaborate with Meta, Google, Hugging Face, PyTorch on open models. We fixed dozens of bugs in Gemma, Llama, Mistral, fixed a gradient accumulation bug, and made finetuning Llama 2x faster & use 70% less VRAM! We value engineers who are proactive, independent and who ship features and ideas quickly - if stuff breaks, that's fine with us! Read more at https://github.com/unslothai/unsloth/wiki#-we-are-hiring