ik_llama.cpp

It works, thanks to https://github.com/ikawrakow/ik_llama.cpp/pull/1911, which got merged in early June. However, there might still be some issue with the chat template.

Author here, it looks like my original comment was flagged for some reason. The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138)

Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish. The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.

Hello, it's the internet calling, today is that day. https://github.com/ikawrakow/ik_llama.cpp Edit: it's gonna be slow if you're not using any VRAM. But it's possible. Software isn't going to speed that up anytime soon, it's just a hardware bandwidth limit.

llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples... When comparing hardware, the output of these tools is very helpful to let others put it into context. The post says the output is "reading speed" but knowing the prefill and token generation speeds would be a lot more helpful.

ikllama.cpp is a fork of llama.cpp which specializes on CPU inference, some benchmarks from 1 year ago: https://github.com/ikawrakow/ik_llama.cpp/discussions/164

Have you tried anything with https://codeberg.org/ikawrakow/illama https://github.com/ikawrakow/ik_llama.cpp and their 4Bit-quants? Or maybe even Microsofts Bitnet? https://github.com/microsoft/BitNet https://github.com/ikawrakow/ik_llama.cpp/pull/337 https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf ? That would be an interesting comparison for running local LLMs on such low-end/edge-devices. Or common office machines with only iGPU.

Take a look at ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp CPU performance is much better than mainline llama, as well as having more quantization types available

I heard that ik_llama.cpp performs better for CPU use: https://github.com/ikawrakow/ik_llama.cpp/

try https://github.com/ikawrakow/ik_llama.cpp

There seems to be some bad blood between ikawrakow and ggerganov: https://github.com/ikawrakow/ik_llama.cpp/discussions/316

There's also some interpersonal conflict in llama.cpp that's hampering other bug fixes https://github.com/ikawrakow/ik_llama.cpp/pull/400