Read the discussion on HN. A full-thread summary is not available yet.
ggml-org
llama.cpp
C++2023–2026
ggml-org / llama.cpp
llama.cpp
LLM inference in C/C++
470Distinct people
737Mentions
5Explicit recommenders
134.8KGitHub stars
What to know before exploring
What it is
LLM inference in C/C++
GitHub
Where it is used
- NVIDIA Developer blog (technical post demonstrating llama.cpp on RTX systems) — NVIDIA (developer blog) documents and demonstrates llama.cpp running and being optimized on NVIDIA RTX systems and describes tooling and applications that leverage llama.cpp for GPU-accelerated local inference. Source
What people are saying
> What's the tok/s you get these days? I ran llama-bench a couple of weeks ago when there was a big speed improvement on llama.cpp…
tarruda used a recommendation phrase. 5 indexed comments match. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- ollama/ollama · linked in 17 of the same discussions
- lostruins/koboldcpp · linked in 6 of the same discussions
- oobabooga/textgen · linked in 4 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:17:07 AM UTC.
The discussion centers on the release of llama.cpp v1 and recent updates to KV cache quantization strategies for coding tasks. Users note a shift in advice regarding whether to keep KV caches at full precision or use 8-bit formats like f8_e4m3 and q8_0.
2 readable replies · Supporting reply 1 · Supporting reply 2Read the discussion on HN. A full-thread summary is not available yet.
Showing the three threads with the most distinct people linking this repo, from 5 threads this week.
Mentions over time
HN mentions by year737 dated mentions
Observed mentions: 2023-03-10 20:01:54 UTC to 2026-10-28 22:30:22 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 316 | 48 | 268 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 156 | 19 | 137 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 124 | 17 | 107 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 141 | 12 | 129 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
Visit https://llama.app and follow the instructions
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
Built-in web UI against llama serve
DescriptionThe main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.Supported backends
Backend
Target devices
BLAS
All
BLIS
All
CANN
Ascend NPU
CUDA
Nvidia GPU
HIP
AMD GPU
Hexagon
Snapdragon
IBM zDNN
IBM Z & LinuxONE
MUSA
Moore Threads GPU
Metal
Apple Silicon
OpenCL
Adreno GPU
OpenVINO [In Progress]
Intel CPUs, GPUs, and NPUs
RPC
All
SYCL
Intel GPU
VirtGPU
VirtGPU APIR
Vulkan
GPU
WebGPU
All
ZenDNN
AMD CPU
Documentation
Tools
cli
completion
server
GBNF grammars
Development
How to build
Running on Docker
Build on Android
Multi-GPU usage
Performance troubleshooting
GGML tips & tricks
XCFramework
Completions
Models
Release process
Contributing
Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
Read the CONTRIBUTING.md for more information
Acknowledgements
yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
Read the full README on GitHubWhat Hacker News says
124 sourcesyou have a few 'claude' proxies on github
llama.cpp recently added Anthropic API support
https://github.com/ggml-org/llama.cpp/pull/17570
In MiniMax M2.1: Built for Real-World Complex Tasks, Multi-Language ProgrammingCompletely free, offline note and text formatting using local LLMs.
I write a lot of notes that end up unstructured and hard to reuse and make sense of. So the goal was to build a fast, easy, local-first way to turn rough input into usable text without sending anything to the cloud.
Desktop builds are available for macOS, Windows, and Linux, with optional Windows/Linux CUDA builds for faster inference. Everything runs locally.
Big shout out to https://github.com/ggerganov/llama.cpp along with PyQt6, pdfplumber, pytesseract, and python-docx for making it easy to wire everything together.
It uses an 8Gb Phi-4 model, however there's no reason you couldn't point to any other GGUF models you have.
Still very much a work in progress, any suggestions welcome.
In Show HN: FormAIt – Free, private offline note formatting using LLMsFrom what I can gather, llama.cpp supports Anthropic's message format now[1], so you can use it with Claude Code[2].
[1]: https://github.com/ggml-org/llama.cpp/pull/17570
[2]: https://news.ycombinator.com/item?id=44654145
In LLM Year in ReviewNo, I don't blog. But I just followed the docs for starting an instance on lambda.ai and the llama.cpp build instructions. Both are pretty good resources. I had already setup an SSH key with lambda and the lambda OS images are linux pre-loaded with CUDA libraries on startup.
Here are my lazy notes + a snippet of the history file from the remote instance for a recent setup where I used the web chat interface built into llama.cpp.
I created an instance gpu_1x_gh200 (96 GB on ARM) at lambda.ai.
connected from terminal on my box at home and setup the ssh tunnel.
ssh -L 22434:127.0.0.1:11434 ubuntu@<ip address of rented machine - can see it on lambda.ai console or dashboard>
Started building llama.cpp from source, history:
21 git clone https://github.com/ggml-org/llama.cpp
22 cd llama.cpp
23 which cmake
24 sudo
Read the rest
apt list | grep libcurl
25 sudo apt-get install libcurl4-openssl-dev
26 cmake -B build -DGGML_CUDA=ON
27 cmake --build build --config Release
MISTAKE on 27, SINGLE-THREADED and slow to build see -j 16 below for faster build
28 cmake --build build --config Release -j 16
29 ls
30 ls build
31 find . -name "llama.server"
32 find . -name "llama"
33 ls build/bin/
34 cd build/bin/
35 ls
36 ./llama-server -hf ggml-org/gpt-oss-120b-GGUF -c 0 --jinja
MISTAKE, didn't specify the port number for the llama-server
37 clear;history
38 ./llama-server -hf Qwen/Qwen3-VL-30B-A3B-Thinking -c 0 --jinja --port 11434
39 ./llama-server -hf Qwen/Qwen3-VL-30B-A3B-Thinking.gguf -c 0 --jinja --port 11434
40 ./llama-server -hf Qwen/Qwen3-VL-30B-A3B-Thinking-GGUF -c 0 --jinja --port 11434
41 clear;history
I switched to qwen3 vl because I need a multimodal model for that day's experiment. Lines 38 and 39 show me not using the right name for the model. I like how llama.cpp can download and run models directly off of huggingface.
Then pointed my browser at http//:localhost:22434 on my local box and had the normal browser window where I could upload files and use the chat interface with the model. That also gives you an openai api-compatible endpoint. It was all I needed for what I was doing that day. I spent a grand total of $4 that day doing the setup and running some NLP-oriented prompts for a few hours.
For the occasional local LLM query, running locally probably won't make much of a dent in the battery life, smaller models like mistral-7b can run at 258 tokens/s on an iPhone 17[0].
The reason why local LLMs are unlikely to displace cloud LLMs is memory footprint, and search.
The most capable models require hundreds of GB of memory, impractical for consumer devices.
I run Qwen 3 2507 locally using llama-cpp, it's not a bad model, but I still use cloud models more, mainly due to them having good search RAG.
There are local tools for this, but they don't work as well, this might continue to improve, but I don't think it's going to get better than the API integrations with google/bing that cloud models use.
[0]: https://github.com/ggml-org/llama.cpp/discussions/4508
In Apple's slow AI pace becomes a strength as market grows weary of spendingInteresting enough today I found that https://huggingface.co/openai/gpt-oss-20b when downloaded an running locally and asked:
====
when have you been built?
I’m a GPT‑4‑based model that OpenAI released on March 14 2023.
The underlying training data goes up to early 2023, and the model was trained in the months leading up to that release. If you’re asking about the ChatGPT product that ships the model to users, it went live in November 2022 and has since received updates (GPT‑3.5, GPT‑4, etc.) that keep it current.
====
But when supposedly running it from https://huggingface.co/chat/models/openai/gpt-oss-20b:
====
when have you been built?
I’m a language model created by OpenAI. The current generation (GPT‑4) that powers this chat was first released in March 2023 and has been updated and fine‑tuned up through the end of 2024. My training d
Read the rest
ata runs up to the beginning of June 2025, so I’m built on knowledge available up to that point.
====
And that makes me thinking that although https://huggingface.co/chat claims to be using the models available to public at https://huggingface.co , it doesn't seems to be true and I raised this question here https://huggingface.co/ggml-org/gpt-oss-20b-GGUF/discussions... , https://github.com/huggingface/inference-playground/issues/1... and https://github.com/ggml-org/llama.cpp/discussions/15396#disc... .
I did an experiment while learning about LLMs and llama.cpp consisting in trying to use create a Lua extension to use llama.cpp API to enhance LLMs with agent/RAG written in Lua with simple code to learn the basics and after more than 5 hours chatting with https://aistudio.google.com/prompts/new_chat?model=gemini-3-... (see the scrapped output of the whole session attached) I've got a lot far in terms of learning how to use an LLM to help develop/debug/learn about a topic (in this case agent/RAG with llama.cpp API using Lua).
I'm posting it here just in case it can help others to see and comment/improve it (it was using around 100K tokens at the end and started getting noticeable slow but still very helpful).
You can see the scrapped text for the whole seession here
https://github.com/ggml-org/llama.cpp/discussions/17600
In So you wanna build a local RAG?Deterministic output is a property of the inference framework, not the model. E.g.
https://docs.sglang.ai/advanced_features/deterministic_infer...
https://github.com/ggml-org/llama.cpp/pull/16016
In Is it possible to have an LLM always give a consistent output?Shocked this wasn't already a feature. Bummed they only seem to have JSON Schema and not something more flexible like BNF grammar's, like llama.cpp has for a long time: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
In Structured outputs on the Claude Developer PlatformIt just depends on what you are doing. A green field react app in typescript with a CRUD API behind? The LLMs are a mind blowing assistant and 1000t/s is crazy.
You are doing embedded development or anything else not as mainstream as web dev? LLMs are still useful but no longer mind blowing and often produce hallucinations. You need to read every line of their output. 1000t/s is crazy but no longer always in a good way.
You are doing stuff which the LLMs haven't seen yet? You are on your own.
There is quite a bit of irony in the fact that the devs of llama.cpp barely use AI - just have a look at the development of support for Qwen3-Next-80B [1].
[1] https://github.com/ggml-org/llama.cpp/pull/16095
In Cerebras Code now supports GLM 4.6 at 1000 tokens/secLlama.cpp launches official WebUI for local LLMs
4 points · Open discussionLlamaCPP supports offloading some experts in a MoE model to CPU. The results are very good and even weaker GPUs can run larger models at reasonable speeds.
n-cpu-moe in https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
In Tongyi DeepResearch – open-source 30B MoE Model that rivals OpenAI DeepResearchRespectfully, this is also bullshit for my use case. For me, the M1 purchase was a step up compared to Intel; the rest is diminishing returns for now.
It’s also not true if you care about certain workloads like LLM performance. My biggest concern for example is memory size and bandwidth, and older chips compare quite favorably to new chips where “GPU VRAM size” now differentiates the premium market and becomes a further upsell, making it less cost-effective. :( I can justify $3k for “run a small LLM on my laptop for my job as ML researcher,” but I still can’t justify $10k for “run a larger model on my Mac Studio”
See https://github.com/ggml-org/llama.cpp/discussions/4167#discu...
In My Impressions of the MacBook Pro M4With MoE models like gpt-oss, you can run some layers on the CPU (and some on GPU): https://github.com/ggml-org/llama.cpp/discussions/15396
Mentions 120b is runnable on 8GB VRAM too: "Note that even with just 8GB of VRAM, we can adjust the CPU layers so that we can run the large 120B model too"
In Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptopI usually use unsloth quants, in this case https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-... - the Q4_K_M variant.
On 128GB I would definitely run a larger model, probably with ~10B active parameters. All depends how many tokens per second is comfortable for you.
To get an idea of the speed difference, there is a benchmark page for llama.cpp on Apple silicon here: https://github.com/ggml-org/llama.cpp/discussions/4167
About quant selection:
https://gist.github.com/Artefact2/b5f810600771265fc1e3944228...
And my workaround for 'shortening' prompt processing time: I load the files I want to work on (usually 1-3) into context with the instruction: read the code and wait. And while the LLM is doing the prompt processing I write my instructions of what I want to have done. Usually the LLM is long finished with PP before I am finis
Read the rest
hed with writing instructions. Due to KV caching the LLM then gives almost instantly the answer.
Showing the 15 most recent of 124 sources. Enable JavaScript to search and browse every indexed mention.