llama-swap

There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap

I use llama.cpp w/ llama-swap https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap

Yes that's right. The config is described by the developer here: https://www.reddit.com/r/LocalLLaMA/comments/1rhohqk/comment... And is in the sample config too: https://github.com/mostlygeek/llama-swap/blob/main/config.ex... iiuc MLX quants are not GGUFs for llama.cpp. They are a different file format which you use with the MLX inference server. LM Studio abstracts all that away so you can just pick an MLX quant and it does all the hard work for you. I don't have a Mac so I have not looked into this in detail.

Shouldn't you be using MLX because it's optimised for Apple Silicon? Many user benchmarks report up to 30% better memory usage and up to 50% higher token generation speed: https://reddit.com/r/LocalLLaMA/comments/1fz6z79/lm_studio_s... As the post says, LM Studio has an MLX backend which makes it easy to use. If you still want to stick with llama-server and GGUF, look at llama-swap which allows you to run one frontend which provides a list of models and dynamically starts a llama-server process with the right model: https://github.com/mostlygeek/llama-swap (actually you could run any OpenAI-compatible server process with llama-swap)

Ollama is a ycombinator startup, so I guess they have to find some roi at some point.[1] I personally found Ollama to be an easy way to try out local LLMs and appreciate them for that (and I still use it to download small models on my laptop and phone (via termux)), but I've long switched to llama.cpp + llama-swap[2] on my dev desktop. I download whatever ggufs I want from hugging face and just do `git pull` and `cmake --build build --config Release` from my llama.cpp directory whenever I want to update. 1: https://www.ycombinator.com/companies/ollama 2: https://github.com/mostlygeek/llama-swap

https://github.com/mostlygeek/llama-swap it's a transparent proxy that automatically launches your selected model with your preferred inference server so that you don't need to manually start/stop the server when you want to switch model so, let's say I have configured roo code to use qwen3 30ba3b as the orchestrator and glm4.5 air as coder, roo code would call the proxy server with model "qwen3" when using orchestrator mode and then kill llama.cpp with qwen3 and restart it with "glm4.5air"