llama-cpp-turboquant

Your setup is definitely a monster compared to mine so it should give much better results. I also believe you run dense models with decent results with that setup. Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw): [*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2 [qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 t

Define "small". The other day I managed to get a context of 195k for Qwen3.5-9b Q4_K_M using a llama.cpp fork that supports TurboQuant: https://github.com/TheTom/llama-cpp-turboquant I think you could replicate this with a larger model on your device. Overall with the right quantisations for both the model and KV cache you can get a lot of mileage out of this old hardware. Speed remains the main limitation, as IIRC I was getting ~26-30tps on a 7700S.

.\llama-server.exe -m ..\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -ngl 999 --n-cpu-moe 41 -c 262144 --port 8081 --flash-attn on --cache-type-k turbo4 --cache-type-v turbo3 --no-mmap --mlock --host 0.0.0.0 -t 8 -tb 8 -np 1 Using this llama.cpp fork https://github.com/TheTom/llama-cpp-turboquant and mostly copying from this video https://www.youtube.com/watch?v=8F_5pdcD3HY Haven't had much time to test it other than asking a few questions & changing some HTML in cline so it might be thick as a brick for all I know, but still worth trying

I built TurboQuant+ (https://github.com/TheTom/llama-cpp-turboquant), the llama.cpp implementation of this paper with extensions: asymmetric K/V compression, boundary layer protection, sparse V dequant, and this week weight compression (TQ4_1S) that shrinks models 28-42%% on disk with minimal quality loss. 5k+ stars, 50+ community testers across Metal, CUDA, and AMD HIP. Cool to see the same WHT + Lloyd-Max math applied to vector search. The data-oblivious codebook property is exactly what makes it work for online KV cache compression too. No calibration, no training, just quantize and go. If anyone is running local LLMs and wants to try it: https://github.com/TheTom/turboquant_plus/blob/main/docs/get...

How-To: Model: hf download leuconoe/Qwen3-8B-Instruct-GGUF Qwen3-8B-Q5_K_M-Instruct.gguf --local-dir ~/.models/` TurboQuant: cd ~ git clone https://github.com/TheTom/llama-cpp-turboquant.git cd llama-cpp-turboquant git checkout feature/turboquant-kv-cache Build for M4: cmake -B build \ -DGGML_METAL=ON \ -DGGML_METAL_EMBED_LIBRARY=ON \ -DCMAKE_BUILD_TYPE=Release cmake --build build -j llama-cli command: ~/llama-cpp-turboquant/build/bin/llama-cli \ -m ~/.models/Qwen3-8B-Q5_K_M-Instruct.gguf \ -ngl 999 \ --cache-type-k turbo3 \ --cache-type-v turbo3 \ --chat-template chatml \ -c 524288 \ -n 624 \ -p "Explain TuboQuant"