Bonsai-demo

Please cite a reputable source for this claim, and keep in mind we are talking about Bonsai 2 and not the original Bonsai. Even on the original one, the authors have replied to such issues with working solutions: https://github.com/PrismML-Eng/Bonsai-demo/issues/131

Exact same test executed on my RTX 3060 (12 GiB VRAM, PCIe 3.0 4x slot): [ Prompt: 95.0 t/s | Generation: 26.5 t/s ] (the test's prompt is very short but with longer ones the I get ~200 prompt processing rate, but I was hoping for a better token generation rate...) Am I understanding correctly that no draft model exists (will never exist or just currently does not exist yet)? There is no draft file in Huggingface's repository ( https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/tr... ) and in the file "scripts/download_models.sh" of the demo repository ( https://github.com/PrismML-Eng/Bonsai-demo/blob/main/scripts... ) I see this remark: if [ "$_family" = "bonsai2" ]; then the projector ships in the same repo; Bonsai 2 has no dspark drafter

Where did you get these instructions? They have a demo repo with a setup.sh script: https://github.com/PrismML-Eng/Bonsai-demo The release tag and weight file you suggest doesn’t match what they wrote.

Depending on which model you're running, you might need to use the custom forks. Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

We took a recently released Bonsai 1.7B ternary model from PrismML (https://github.com/PrismML-Eng/Bonsai-demo) and ran our agentic evolution search on it for 6 hours to optimize the Metal kernels. The search was fully autonomous. Measured against unmodified upstream llama.cpp at the same Bonsai/Q2_0 commit, same M4 Max: - tg128: 309.82 → 442.42 t/s (+42.0%) - pp512: 4250.32 → 4622.63 t/s (+8.8%)

I'm very skeptical of the advantage they're claiming here. The whitepaper [0] only compares these to full precision models, when the more interesting (and probably more meaningful) comparison would be with other quantized models with a similar memory footprint. Especially considering that these models seem to more or less just be quantized variants of Qwen3 with custom kernels and other inference optimizations (?) rather than fine tuned or trained from scratch with a new architecture, I am very surprised (or suspicious rather) that they didn't do the obvious comparison with a quantized Qwen3. Their (to my knowledge) new measure/definition of intelligence seems reasonable, but introducing something like this without thorough benchmarking + model comparison is even more of a red flag to me. [0] https://github.com/PrismML-Eng/Bonsai-demo/blo

See page 10: https://github.com/PrismML-Eng/Bonsai-demo/blob/main/1-bit-b...

If you look at their whitepaper (https://github.com/PrismML-Eng/Bonsai-demo/blob/main/1-bit-b...) you'll notice that it does have some tradeoffs due to model intelligence being reduced (page 10) The average of MMLU Redux,MuSR,GSM8K,Human Eval+,IFEval,BFCLv3 for this model is 70.5 compared to 79.3 for Qwen3, that being said the model is also having a 16x smaller size and is 6x faster on a 4090....so it is a tradeoff that is pretty respectable I'd be interested in fine tuning code here personally