Read the discussion on HN. A full-thread summary is not available yet.
ggml-org
llama.cpp
C++2023–2026
ggml-org / llama.cpp
llama.cpp
LLM inference in C/C++
470Distinct people
737Mentions
5Explicit recommenders
134.8KGitHub stars
What to know before exploring
What it is
LLM inference in C/C++
GitHub
Where it is used
- NVIDIA Developer blog (technical post demonstrating llama.cpp on RTX systems) — NVIDIA (developer blog) documents and demonstrates llama.cpp running and being optimized on NVIDIA RTX systems and describes tooling and applications that leverage llama.cpp for GPU-accelerated local inference. Source
What people are saying
> What's the tok/s you get these days? I ran llama-bench a couple of weeks ago when there was a big speed improvement on llama.cpp…
tarruda used a recommendation phrase. 5 indexed comments match. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- ollama/ollama · linked in 17 of the same discussions
- lostruins/koboldcpp · linked in 6 of the same discussions
- oobabooga/textgen · linked in 4 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:17:07 AM UTC.
The discussion centers on the release of llama.cpp v1 and recent updates to KV cache quantization strategies for coding tasks. Users note a shift in advice regarding whether to keep KV caches at full precision or use 8-bit formats like f8_e4m3 and q8_0.
2 readable replies · Supporting reply 1 · Supporting reply 2Read the discussion on HN. A full-thread summary is not available yet.
Showing the three threads with the most distinct people linking this repo, from 5 threads this week.
Mentions over time
HN mentions by year737 dated mentions
Observed mentions: 2023-03-10 20:01:54 UTC to 2026-10-28 22:30:22 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 316 | 48 | 268 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 156 | 19 | 137 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 124 | 17 | 107 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 141 | 12 | 129 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
Visit https://llama.app and follow the instructions
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
Built-in web UI against llama serve
DescriptionThe main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.Supported backends
Backend
Target devices
BLAS
All
BLIS
All
CANN
Ascend NPU
CUDA
Nvidia GPU
HIP
AMD GPU
Hexagon
Snapdragon
IBM zDNN
IBM Z & LinuxONE
MUSA
Moore Threads GPU
Metal
Apple Silicon
OpenCL
Adreno GPU
OpenVINO [In Progress]
Intel CPUs, GPUs, and NPUs
RPC
All
SYCL
Intel GPU
VirtGPU
VirtGPU APIR
Vulkan
GPU
WebGPU
All
ZenDNN
AMD CPU
Documentation
Tools
cli
completion
server
GBNF grammars
Development
How to build
Running on Docker
Build on Android
Multi-GPU usage
Performance troubleshooting
GGML tips & tricks
XCFramework
Completions
Models
Release process
Contributing
Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
Read the CONTRIBUTING.md for more information
Acknowledgements
yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
Read the full README on GitHubWhat Hacker News says
316 sourcesHi, I haven't tried this myself, but it seems there's a way?
https://github.com/ggerganov/llama.cpp/blob/master/examples/...
The call takes a "stream" boolean:
stream: It allows receiving each predicted token in real-time instead of waiting for the completion to finish. To enable this, set to true.
And the response includes:
stop: Boolean for use with stream to check whether the generation has stopped (Note: This is not related to stopping words array stop from input options)
Certainly the local web interface has a stop button, and I'm pretty sure that one did work.
But maybe I'm misunderstanding the challenge here?
In Emacs-copilot: Large language model code completion for EmacsAuthor here. It does make use of mmap(). I worked on adding mmap() support to llama.cpp back in March, specifically so I could build things like Emacs Copilot. See: https://github.com/ggerganov/llama.cpp/pull/613 Recently I've been working with Mozilla to create llamafile, so that using llama.cpp can be even easier. We've also been upstreaming a lot of bug fixes too!
In Emacs-copilot: Large language model code completion for EmacsInstructions:
1. Install llama.cpp from https://github.com/ggerganov/llama.cpp. Alternatively, install https://github.com/oobabooga/text-generation-webui
2. Go to https://huggingface.co/TheBloke and search for GGUF. Download and put the model file in the same directory. Then find the "example llama.cpp command line" and run without the "-ngl 35" switch.
However, at this point, if your laptop has at least 32 GB of RAM, there is no point in trying anything except Mixtral 8x7b and its fine-tunes. It is fast (4 tokens per second on an 8-core Ryzen without any GPU acceleration - which would not work on integrated Ryzen APUs anyway because they don't have dedicated VRAM) and provides answers only slightly worse than ChatGPT 3.5. Its main deficiency is the tendency to forget the initial instruction - for example, when asked to explain a particul
Read the rest
ar SAMBA configuration file, it started OK, but then continued mentioning directives that were not in the file under discussion.
You can find an exception to any rule, but you'll find, in general, that its both self-evident and unsurprising that make is on the decline, and being replaced by other tools.
plan9 exists in an enviable position of only needing to build in a few controlled environments.
For example, http://9p.io/sources/plan9/sys/src/libmach/mkfile reads:
CFLAGS=$CFLAGS -I/sys/src/cmd
Cute. Non-portable. A perfect example of how make is useful in trivial or contrived circumstances only.
/shrug
Try reading the llama.cpp makefile, which is an example of an excellent cross platform makefile that works very well: https://github.com/ggerganov/llama.cpp/blob/master/Makefile
^ this is what many real Makefiles look like, and this one is excellent and well maintained.
Many are worse.
Make is good when you only have easy problems to solve (like, specifically,
Read the rest
only having to build one specific constrained environment for one specific compiler).
This article seems very corporate centric. Like, I am able to run a ChatGPT3-ish code LLM locally on a 2015 midrange laptop. Just like this:
wget https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/dee…
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins
Haven't people realized what they can run themself?
In 2023: The Year of AICosmopolitan is a nice hack but I'd never use it.
I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped cross-platform software and it's a one time trivial cost to prepare a multi OS pipeline. The actual problems come from dependencies and I'm sure you'll have to solve the same problems with cosmopolitan.
Llamafile is another hack I'd never use - why limit myself to the models supported by llamafile when I can just install llama.cpp? (the web interface is also part of llama.cpp https://github.com/ggerganov/llama.cpp/pull/1998)
Moreover, people with GPUs will likely want to use something with better performance than llama.cpp (gptq or exllama2)
In Llamafile: Bringing LLMs to the people, and to your own computerCUDA: Faster Mixtral Prompt Processing
3 points · Open discussionI haven't used the llama2 models much in quite awhile, because they just aren't very good compared to other options that exist at this point. The instruction-tuned variants of Mistral and Mixtral seem to have very little trouble responding in JSON when I ask for it. However, with LLMs that you run yourself, you can also enforce a grammar for the response if you want to, guaranteeing that it will respond with valid JSON (that matches your schema!) and no extraneous text.
Something potentially helpful here: https://github.com/ggerganov/llama.cpp/discussions/2494
If you fine-tuned a base model (like the one in the article) on various inputs and the expected JSON output for each input, it would probably do even better.
In Mistral 7B Fine-Tune OptimizedIn this case they're comparing against llama.cpp because the code is literally a modification of llama.cpp. I'm not talking about using the ggml lib for matrix calculations, it's literally using the llama.cpp main.cpp and other normal llama.cpp code. It's a fork. It is directly comparable.
https://github.com/ggerganov/llama.cpp/pull/4543 [Review] Merge PowerInfer with llama.cpp mainline #4543
https://github.com/ggerganov/llama.cpp/discussions/4534#disc... "The x11 speedup is kind of cherrypicked because the llama.cpp GPU code for Falcon 40b is just not well-optimized."
In High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUsQ4 comes out to be ~26GB but Apple doesn't let you load it on a 32GB Mac machine because they put a limit on the max usable unified memory at ~21GB (`device. recommendedMaxWorkingSetSize`) [1]. So for Q4 Mixtral MoE you'd need a 64GB Mac machine unfortunately.
Unless you use this hack [2].
[1] https://developer.apple.com/forums/thread/732035
[2] https://github.com/ggerganov/llama.cpp/discussions/2182#disc...
In LLM in a Flash: Efficient LLM Inference with Limited MemoryThis is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: https://github.com/ggerganov/llama.cpp/pull/4543
One thing that caught my interest is this line from their readme:
> PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers.
Apple's Metal/M3 is perfect for this because CPU and GPU share memory. No need to do any data transfers. Checkout mlx from apple: https://github.com/ml-explore/mlx
In PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]Sure, this is common e.g. https://github.com/ggerganov/llama.cpp/pull/613
In LLM in a Flash: Efficient LLM Inference with Limited MemoryPerformance of llama.cpp on Apple Silicon A-series
100 points · Open discussionShowing the 15 most recent of 316 sources. Enable JavaScript to search and browse every indexed mention.