The discussion centers on the release of llama.cpp v1 and recent updates to KV cache quantization strategies for coding tasks. Users note a shift in advice regarding whether to keep KV caches at full precision or use 8-bit formats like f8_e4m3 and q8_0.
2 readable replies · Supporting reply 1 · Supporting reply 2ggml-org
llama.cpp
C++2023–2026
ggml-org / llama.cpp
llama.cpp
LLM inference in C/C++
470Distinct people
737Mentions
5Explicit recommenders
134.8KGitHub stars
What to know before exploring
What it is
LLM inference in C/C++
GitHub
Where it is used
- NVIDIA Developer blog (technical post demonstrating llama.cpp on RTX systems) — NVIDIA (developer blog) documents and demonstrates llama.cpp running and being optimized on NVIDIA RTX systems and describes tooling and applications that leverage llama.cpp for GPU-accelerated local inference. Source
What people are saying
> What's the tok/s you get these days? I ran llama-bench a couple of weeks ago when there was a big speed improvement on llama.cpp…
tarruda used a recommendation phrase. 5 indexed comments match. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- ollama/ollama · linked in 17 of the same discussions
- lostruins/koboldcpp · linked in 6 of the same discussions
- oobabooga/textgen · linked in 4 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.
Read the discussion on HN. A full-thread summary is not available yet.
Commenters are divided on whether Apple's new local AI app will succeed by leveraging hardware integration and privacy or fail due to DRAM supply constraints and inferior performance compared to cloud subscriptions. The strongest disagreement centers on the viability of local inference for mainstream users, with some citing high hardware costs and usability gaps while others predict Apple's vertical integration will eventually dominate.
551 readable replies · Supporting reply 1 · Supporting reply 2 · Supporting reply 3 · Supporting reply 4Showing the three threads with the most distinct people linking this repo, from 5 threads this week.
Mentions over time
HN mentions by year737 dated mentions
Observed mentions: 2023-03-10 20:01:54 UTC to 2026-10-28 22:30:22 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 316 | 48 | 268 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 156 | 19 | 137 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 124 | 17 | 107 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 141 | 12 | 129 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
Visit https://llama.app and follow the instructions
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
Built-in web UI against llama serve
DescriptionThe main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.Supported backends
Backend
Target devices
BLAS
All
BLIS
All
CANN
Ascend NPU
CUDA
Nvidia GPU
HIP
AMD GPU
Hexagon
Snapdragon
IBM zDNN
IBM Z & LinuxONE
MUSA
Moore Threads GPU
Metal
Apple Silicon
OpenCL
Adreno GPU
OpenVINO [In Progress]
Intel CPUs, GPUs, and NPUs
RPC
All
SYCL
Intel GPU
VirtGPU
VirtGPU APIR
Vulkan
GPU
WebGPU
All
ZenDNN
AMD CPU
Documentation
Tools
cli
completion
server
GBNF grammars
Development
How to build
Running on Docker
Build on Android
Multi-GPU usage
Performance troubleshooting
GGML tips & tricks
XCFramework
Completions
Models
Release process
Contributing
Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
Read the CONTRIBUTING.md for more information
Acknowledgements
yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
Read the full README on GitHubWhat Hacker News says
737 sourcesllama.cpp looks to be cloning openai's realtime websocket API with this audio model as its first use case? I guess this model is either the first speech-to-speech of its kind, OR it's the first good-enough one. I'm thinking I need to give this more testing!
- https://github.com/ggml-org/llama.cpp/pull/31287
- https://github.com/ggml-org/llama.cpp/pull/31269
In Introducing LFM3: Every Modality for Specialized IntelligenceYes. All you need is a formal grammar that defines the syntax of the text being generated. The llama.cpp repo has a decent writeup on the way that grammar-constrained decoding works in general [0], and it includes a bit of a simplistic example for generating syntactically-correct C code [1]. No reason this couldn't be extended to other things -- I personally have run experiments with generating syntactically-correct SQL statements when doing text-to-sql [2], and I think it's pretty cool.
[0] - https://github.com/ggml-org/llama.cpp/tree/master/grammars
[1] - https://github.com/ggml-org/llama.cpp/blob/master/grammars/c...
[2] - https://gist.github.com/HanClinto/b853571136fd42f8e2bde4c99c...
In Fixing Tool-Call Accuracy in vLLMExagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.
https://github.com/ggml-org/llama.cpp/issues/27330
In Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware> Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpp https://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
I mean, yes, that works at dev time; but going from zero to nonzero local inference components makes an agent implementation suddenly much more complex to ship to production / drastically narrows the set of hosts such an implementation can be deployed onto.
In We’re releasing Jev 1.14.0 along with experimental features for dreamersThe extra funny thing about this is that the target audience is people who use agentic AI systems (system one thinking as a compliment to system two thinking). Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpp https://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
The main benefit of Jev seems to be broader knowledge? But local models (especially the ones actually trained specifically for decisions, rather than ones that "support" it like embeddinggemma) are probably good enough for what most people use it for. They also seem to be quite easy to fine tune (I haven't tried, but seems to be what people say).
In We’re releasing Jev 1.14.0 along with experimental features for dreamers> https://github.com/darshi1337/apogee/blob/main/MODELS.md#loc...
> Unlike Ollama, [llama.cpp] serves one model at a time: the GGUF you launched it with.
FWIW: llama-server's supported multiple models for a while now:
https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...
In Show HN: Apogee: Rebuilding Mozilla's Orbit, fully local and privatehttps://en.wikipedia.org/wiki/List_of_AMD_graphics_processin...
https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
If you know what any given model uses for BW per token and flops per token, you can roughly infer for many models until you get better benchmarks.
There is also:
CUDA: https://github.com/ggml-org/llama.cpp/discussions/15013
HIP: https://github.com/ggml-org/llama.cpp/discussions/15021
Vulkan: https://github.com/ggml-org/llama.cpp/discussions/10879
Metal: https://github.com/ggml-org/llama.cpp/discussions/4167
These three are measured against the old model of Llama 2 7B, but it also means the numbers are without any per-model quirks that reduce compute and memory BW usage, thus more or less generically applicable.
In How many GPUs is 1M/B/T tokens?They do make their standards explicit: https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTI...
What part do you think is irrational gatekeeping?
In Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/sThe docs are a great start.
https://github.com/ggml-org/llama.cpp/blob/master/docs/backe...
In Show HN: Janus – Go binary that runs GGUF models via Vulkan on AMD/Intel/NvidiaIt's being worked on for upstream: https://github.com/ggml-org/llama.cpp/issues/29058
In Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller FootprintIt looks like upstream work is underway: https://github.com/ggml-org/llama.cpp/issues/29058
The sheer number of geniuses working so fast on this repo never ceases to amaze me.
In Best LLM for every budget, updated dailyI haven't used llama.cpp for image generation either but I recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is still limited to text.
In Qwen-Image-2.1: Compact, efficient, and unified image creationlike https://github.com/ggml-org/llama.cpp/tree/master/examples/d... ?
At the risk of stating the obvious llama.cpp isn't just about LLaMa as https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... someone else pointed out.
In Qwen Image 2.1Not all architectures are supported by llama.cpp . The GGUF format encodes the NN in a standardized way, but then you need code that can use that NN structure.
I understand that llama.cpp could only output text, last time I checked (I do not know how to find a good source for that though).
See https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... , the
enum llm_arch {
...
In Qwen Image 2.1it's multimodal, see https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...
In Qwen-Image-2.1: Compact, efficient, and unified image creationShowing the 15 most recent of 737 sources. Enable JavaScript to search and browse every indexed mention.