Read the discussion on HN. A full-thread summary is not available yet.
ggml-org
llama.cpp
C++2023–2026
ggml-org / llama.cpp
llama.cpp
LLM inference in C/C++
470Distinct people
737Mentions
5Explicit recommenders
134.8KGitHub stars
What to know before exploring
What it is
LLM inference in C/C++
GitHub
Where it is used
- NVIDIA Developer blog (technical post demonstrating llama.cpp on RTX systems) — NVIDIA (developer blog) documents and demonstrates llama.cpp running and being optimized on NVIDIA RTX systems and describes tooling and applications that leverage llama.cpp for GPU-accelerated local inference. Source
What people are saying
> What's the tok/s you get these days? I ran llama-bench a couple of weeks ago when there was a big speed improvement on llama.cpp…
tarruda used a recommendation phrase. 5 indexed comments match. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- ollama/ollama · linked in 17 of the same discussions
- lostruins/koboldcpp · linked in 6 of the same discussions
- oobabooga/textgen · linked in 4 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:17:07 AM UTC.
The discussion centers on the release of llama.cpp v1 and recent updates to KV cache quantization strategies for coding tasks. Users note a shift in advice regarding whether to keep KV caches at full precision or use 8-bit formats like f8_e4m3 and q8_0.
2 readable replies · Supporting reply 1 · Supporting reply 2Read the discussion on HN. A full-thread summary is not available yet.
Showing the three threads with the most distinct people linking this repo, from 5 threads this week.
Mentions over time
HN mentions by year737 dated mentions
Observed mentions: 2023-03-10 20:01:54 UTC to 2026-10-28 22:30:22 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 316 | 48 | 268 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 156 | 19 | 137 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 124 | 17 | 107 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 141 | 12 | 129 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
Visit https://llama.app and follow the instructions
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
Built-in web UI against llama serve
DescriptionThe main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.Supported backends
Backend
Target devices
BLAS
All
BLIS
All
CANN
Ascend NPU
CUDA
Nvidia GPU
HIP
AMD GPU
Hexagon
Snapdragon
IBM zDNN
IBM Z & LinuxONE
MUSA
Moore Threads GPU
Metal
Apple Silicon
OpenCL
Adreno GPU
OpenVINO [In Progress]
Intel CPUs, GPUs, and NPUs
RPC
All
SYCL
Intel GPU
VirtGPU
VirtGPU APIR
Vulkan
GPU
WebGPU
All
ZenDNN
AMD CPU
Documentation
Tools
cli
completion
server
GBNF grammars
Development
How to build
Running on Docker
Build on Android
Multi-GPU usage
Performance troubleshooting
GGML tips & tricks
XCFramework
Completions
Models
Release process
Contributing
Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
Read the CONTRIBUTING.md for more information
Acknowledgements
yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
Read the full README on GitHubWhat Hacker News says
156 sourcesLlama v3.3 70B after quantization runs reasonably well on a 24GB GPU (7900XTX or 4090) and 64GB of regular RAM. Software: https://github.com/ggerganov/llama.cpp .
In GPT-5 is behind scheduleCompare performance on various Macs here as it gets updated:
https://github.com/ggerganov/llama.cpp/discussions/4167
OMM, Llama 3.3 70B runs at ~7 text generation tokens per second on Macbook Pro Max 128GB, while generating GPT-4 feeling text with more in depth responses and fewer bullets. Llama 3.3 70B also doesn't fight the system prompt, it leans in.
Consider e.g. LM Studio (0.3.5 or newer) for a Metal (MLX) centered UI, include MLX in your search term when downloading models.
Also, do not scrimp on the storage. At 60GB - 100GB per model, it takes a day of experimentation to use 2.5TB of storage in your model cache. And remember to exclude that path from your TimeMachine backups.
In Phi-4: Microsoft's Newest Small Language Model Specializing in Complex ReasoningLlama.cpp Now Supports Qwen2-VL (Vision Language Model)
155 points · Open discussionHi, just wanted to say how much I appreciate your work.
I'm curious if you have considered implementing Microsoft's Guidance (https://github.com/guidance-ai/guidance)? Their approach offers significant speed improvements, which I understand can sometimes be shortcoming of GBNF (e.g https://github.com/ggerganov/llama.cpp/issues/4218).
In Structured Outputs with OllamaIf you want avoid startup cost, llguidance [0] has no compilation phase and by far the fullest JSON support [1] of any library. I did a PoC llama.cpp integration [2] though our focus is mostly server-side [3].
[0] https://github.com/guidance-ai/llguidance
[1] https://github.com/guidance-ai/llguidance/blob/main/parser/s...
[2] https://github.com/ggerganov/llama.cpp/pull/10224
[3] https://github.com/guidance-ai/llgtrt
In Structured Outputs with OllamaIf anyone needs a more powerful constrain outputs, llama.cpp support gbnf:
https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
In Structured Outputs with OllamaNice.
That said... I mean...
> The journey to integrate K/V context cache quantisation into Ollama took around 5 months.
??
They incorrectly tagged #7926 which is a 2 line change, instead of #6279 where it was implemented, which made me dig a bit deeper and reading the actual change it seems:
The commit (1) is:
> params := C.llama_context_default_params()
> ...
> params.type_k = kvCacheTypeFromStr(strings.ToLower(kvCacheType)) <--- adds this
> params.type_v = kvCacheTypeFromStr(strings.ToLower(kvCacheType)) <--- adds this
Which has been part of llama.cpp since Dev 7, 2023 (2).
So... mmmm... while this is great, somehow I'm left feeling kind of vaguely put-off by the comms around what is really 'we finally support some config flag from llama.cpp that's been there for really quite a long time'.
> It took 5 months, but we g
Read the rest
ot there in the end.
... I guess... yay? The challenges don't seem like they were technical, but I guess, good job getting it across the line in the end?
[1] - https://github.com/ollama/ollama/commit/1bdab9fdb19f8a8c73ed...
[2] - since https://github.com/ggerganov/llama.cpp/commit/bcc0eb4591bec5...
re Temperature config option: I've found it useful for trying to generate something akin to a sampling-based confidence score for chat completions (e.g., set the temperature a bit high, run the model a few times and calculate the distribution of responses). Otherwise haven't figured out a good way to get confidence scores in llama.cpp (Been tracking this git request to get log_probs https://github.com/ggerganov/llama.cpp/issues/6423)
In Llama.cpp guide – Running LLMs locally on any hardware, from scratchFYI, if you're on Ubuntu 24.04, it's easy to build llama.cpp with AMD ROCm GPU acceleration. Debian enabled support for a wider variety of hardware than is available in the official AMD packages, so this should work for nearly all discrete AMD GPUs from Vega onward (with the exception of MI300, because Ubuntu 24.04 shipped with ROCm 5):
sudo apt -y install git wget hipcc libhipblas-dev librocblas-dev cmake build-essential
# add yourself to the video and render groups
sudo usermod -aG video,render $USER
# reboot to apply the group changes
# download a model
wget --continue -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf \
https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin…
# build llama.cpp
git clone https://github.com/ggerganov/llam
Read the rest
a.cpp.git
cd llama.cpp
git checkout b3267
HIPCXX=clang++-17 cmake -S. -Bbuild \
-DGGML_HIPBLAS=ON \
-DCMAKE_HIP_ARCHITECTURES="gfx803;gfx900;gfx906;gfx908;gfx90a;gfx1010;gfx1030;gfx1100;gfx1101;gfx1102" \
-DCMAKE_BUILD_TYPE=Release
make -j8 -C build
# run llama.cpp
build/bin/llama-cli -ngl 32 --color -c 2048 \
--temp 0.7 --repeat_penalty 1.1 -n -1 \
-m ../dolphin-2.2.1-mistral-7b.Q5_K_M.gguf \
--prompt "Once upon a time"
I think this will also work on Rembrandt, Renoir, and Cezanne integrated GPUs with Linux 6.10 or newer, so you might be able to install the HWE kernel to get it working on that hardware.
With that said, users with CDNA 2 or RDNA 3 GPUs should probably use the official AMD ROCm packages instead of the built-in Ubuntu packages, as there are performance improvements for those architectures in newer versions of rocBLAS.
You can get a release binary from https://github.com/ggerganov/llama.cpp/releases too.
In Llama.cpp guide – Running LLMs locally on any hardware, from scratchI am happily using qwen2.5-coder-7b-instruct-q3_k_m.gguf with a context size of 32768 on an RTX 3060 Mobile with 6GB VRAM using llama.cpp [2]. With 16GB VRAM, you could use qwen2.5-7b-instruct-q8_0.gguf which is basically indistinguishable from the fp16 variant.
[1] https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF
[2] https://github.com/ggerganov/llama.cpp
In OpenCoder: Open Cookbook for Top-Tier Code Large Language ModelsNo need to use Ollama. LLama.cpp has its own OpenAI-compatible server[0] and it works great.
[0] https://github.com/ggerganov/llama.cpp#web-server
In Go library for in-process vector search and embeddings with llama.cppI’ve only toyed with them a bit, and had a similar experience - but did find I got better output by forcing them to adhere to a fixed grammar: https://github.com/ggerganov/llama.cpp/tree/master/grammars
For context, I was playing with a script to bulk download podcasts, transcribe with whisper, pass the transcription to llama.cpp to ID ads, then slice the ads out with ffmpeg. I started with the generic json_array example grammar, then iteratively tweaked it.
In Quantized Llama models with increased speed and a reduced memory footprintShowing the 15 most recent of 156 sources. Enable JavaScript to search and browse every indexed mention.