amd-strix-halo-toolboxes

No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp. These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...

yes. I build a docker file from this: https://github.com/kyuz0/amd-strix-halo-toolboxes It has all the backends you want. Just a script rebuilds it. Then I just put it in a docker compose and whenever we want to update it, we pull the repo and rebuild it. I just a models.ini like: version = 1 ``` [*] n-gpu-layers = 999 threads = 28 flash-attn = on mlock = off mmap = off fit = off warmup = on jinja = true direct-io = on cache-prompt = true cache-reuse = 256 models-max = 1 models-autoload = 1 timeout = 600 ctx-checkpoints = 256 cache-ram = 0 mlock = on kv-unified = on no-context-shift = on swa-full = on [Q36-35B-A3B] model = Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf mmproj = mmproj-BF16.gguf rope-scaling = yarn rope-scale = 1.14441 yarn-orig-ctx = 262144 override-kv = qwen35.context_length=int:1000000 ctx-size = 300000 reasoning = auto reasoning-bud

Sounds like you got it sorted, but more generally this may be interesting: https://github.com/kyuz0/amd-strix-halo-toolboxes

If you just want to run models, most of TFA is taking the scenic route. All you really need is podman, toolbx, and the Strix Halo toolbox images from https://github.com/kyuz0/amd-strix-halo-toolboxes. Then you just download your ggufs and hand them to llama-server. Yes, there are other solutions that are a bit more hand-holdy, but if you already know how to use docker/podman and just want to get something working in an evening, this works too.

I realize it does not address the OP security concerns, but I'm having success running rocm containers[0] on alpine linux specifically for llama.cpp. I also got vLLM to run in a rocm container, but I didn't have time to to diagnose perf problems, and llama.cpp is working well for my needs. [0] https://github.com/kyuz0/amd-strix-halo-toolboxes

I took my setup from here: https://github.com/kyuz0/amd-strix-halo-toolboxes Still lot to learn, but after a while you have something like Qwen3-Coder-Next-Q8_0 running and - at least for me - it works quite well, both as ChatGPT like chat-interface using llama.cpp and as coding agent

If you're on Strix Halo, check out Donato's prebuilt toolboxes for ROCm with RADV or Vulkan: https://github.com/kyuz0/amd-strix-halo-toolboxes It takes all the work out of it, you just start llama-server in the container context and you're off doing inference without having to figure out dependencies.

I have AMD Strix Halo (395) on my work laptop (HP Ultrabook G1A) as well as at home with Framework Desktop. On both i have setup lemonade-server on system start. At work i use Qwen3 Coder 30B-3A with continue.dev. It serves me well in 90% of cases. At home i have 128GB RAM. I try a bit GPT120B. I host Open WebUI on it and connect via https and wireguard to it, so i can use it as PWA on my phone. I love not needing to think about where my data goes. But i would like to allow parallel requests, so i need to tinker a bit more. Maybe llama-swap is enough. I just need to see how to deal with context length. My models stop or go into infinite loop after some messages. But then i often start a new chat. Lemonade-server runs with llama.cpp, vllm seems to be scaling better thoug, but is not so easy to set up. Unsloth GGUFs are great resource for m