llama.cpp

I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware): git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j Then downloaded & verified `Ternary-Bonsai-2-27B-PTQ1_0.gguf` from HF and ran ./llama.cpp/build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 32768 -b 256 -ub 64 --temp 1.0 --top-p 0.95 --top-k 20 -ctk q8_0 -ctv q8_0 -p "hello world in x86 assembler" -n 256 the parameters were suggested by gpt-5.6-luna to reduce memory footprint, as the defaults ran OOM on my gpu. result looks good: [ Prompt: 165.6 t/s | Generation: 40.6 t/s ] would be nice if they upstreamed their changes so that it runs wit

If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-... This should work: cd /tmp # Get the Prism macOS runtime curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/ll… -o bonsai-runtime.tar.gz tar -xzf bonsai-runtime.tar.gz # Get the ~5.95 GB GGUF model: curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bon… -o Ternary-Bonsai-2-27B-PTQ1_0.gguf # Run the server, I used port 8331 ./llama-prism-b10685-7dffb15/llama-server \ -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \ --port 8331 -ngl 99 -fa on -c 327

I got their previous model working in their custom fork of llama.cpp (https://github.com/PrismML-Eng/llama.cpp). I haven't tried this one yet, but will find some time to benchmark it sometime this week. Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.

Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \ --threads-batch 8 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --log-colors on # llama.cpp is forked one: https://github.com/PrismML-Eng/llama.cpp.git # The server can serve 5 parallel request, with each request capped at around `13K` tokens... # A bit of of benchmarks I did: # 1. Input: 1001 tokens, ttfs: 0.3 second, outputs: 1618 tokens ~140t/s

prismML provides a llama.cpp fork which is compatible with the 1 bit models: https://github.com/PrismML-Eng/llama.cpp After fails with Ollama and main llama.cpp the fork worked on my M5 MBA. Edit: Typos

I have older M1 air with 8GB, but still getting ober 23 t/s on 4B model.. and the quality of outputs is on par with top models of similar size. 1. Clone their forked repo: `git clone https://github.com/PrismML-Eng/llama.cpp.git` 2. Then (assuming you already have xcode build tools installed): cd llama.cpp cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j$(sysctl -n hw.logicalcpu) 3. Finally, run it with (you can adjust arguments): ./build/bin/llama-server -m ~/Downloads/Bonsai-8B.gguf --port 80 --host 0.0.0.0 --ctx-size 0 --parallel 4 --flash-attn on --no-perf --log-colors on --api-key some_api_key_string Model was first downloaded from: https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main