HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
LostRuins
koboldcpp
C++2023–2026
LostRuins / koboldcpp

koboldcpp

KoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install.

AI & Machine Learning
Explore on GitHubRead Hacker News discussions
18Distinct people
34Mentions
0Explicit recommenders
12.3KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

KoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 12:45:45 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year34 dated mentions

Observed mentions: 2023-04-07 15:35:51 UTC to 2026-06-28 05:44:05 UTC.

051015202023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 19 mentions; 1 submissions, 18 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 8 mentions; 1 submissions, 7 comments20242025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 4 mentions; 1 submissions, 3 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC19118
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC817
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC312
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC413

What Hacker News says

19 sources
KronisLV·Nov 17, 2023
> The solution will (I hope) ultimately be open-source LLMs that you can run locally, on your own hardware. There are projects like llama.cpp https://github.com/ggerganov/llama.cpp and koboldcpp: https://github.com/LostRuins/koboldcpp that can make running open source models pretty easy. As for where to find the models, you can look at HuggingFace for quite the selection, depending on what your needs are and what your hardware (CPU and RAM or GPU and VRAM, or both) can run: https://huggingface.co/models I was actually curious about this, so I wrote a blog post on it: https://blog.kronis.dev/tutorials/self-hosting-an-ai-llm-cha... which wasn't anything too serious, in addition to formats like GGUF I think becoming more popular than GGML since. Things are moving ahead quite rapidly over there. I guess it can also be easier to pay for somet
Read the rest
hing like NovelAI for those who want something for general purpose writing (including most NSFW stuff), but it wasn't exactly amazing for my needs. Despite the name, it wasn't great at prose enhancement: to take a dry sounding paragraph or two and rewrite it until I've found an additional expression or two to include instead, or a few sentence structures than work better. Kind of fun as a replacement for AI Dungeon or something like Character.AI, though. Then again, the local models need expensive hardware to actually run well, if you want good results (somewhere above 20B parameters from what I've seen). I do wonder how much better the models that approach 200B would be for general text generation use cases, if the performance would actually be good. Even more so, when it comes to writing code, or all of the boring stuff like boilerplate/scaffolding code, utility functions, or even stubs for tests. ChatGPT, Phind and GitHub Copilot are all there, but could be even better.
In A failed AI girlfriend product, and my lessons
narrator·Nov 16, 2023
I got koboldcpp (https://github.com/LostRuins/koboldcpp) running locally on my M1 mac with 16gb. Use the guides on http://4chan.org/g/ in the AI Chatbot Thread. In the future, you will own and control access to your very own AI girlfriend running on your local machine, and you will be happy. Think of all the millions of guys in China and elsewhere who, due to sex selective abortion, will never have a wife. They deserve to have their own Waifu running on their own computer that serves them alone, and not the interests of the state or some corporation.
In A failed AI girlfriend product, and my lessons
brucethemoose2·Oct 31, 2023
> We can achieve up to 100 tokens per second single-stream while GPT-4 runs around 20 tokens per second at best. Is that with batching? If so, thats quite impressive. > certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4. Some of this is sampler tuning. Y'all should look at grammar based sampling (https://github.com/ggerganov/llama.cpp/pull/1773) if you aren't using it already, as well as some of the "dynamic" sampling like mirostat and dynatemp: https://github.com/LostRuins/koboldcpp/pull/464 I think these should work with nvidia's implementation if you just swap the sampling out with the HF version. BTW, all this is a great advantage of pulling away from OpenAI. You can dig in and implement experimental features that you just can
Read the rest
't necessarily do through their API.
In Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
brucethemoose2·Oct 31, 2023
If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible: https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment. I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.
In Mistral 7B,The complete Guide of the Best 7B model
brucethemoose2·Oct 9, 2023
It works like this: - The AI Horde hosts a web app (Kobold Lite) geared towards LLM chat/instruct. Its mature, predating LLAMA and GPT 3.5 and largely developed when the RP community was running GPT-J finetunes. There are desktop apps that can access this API as well. - The user sets the chat syntax/format and picks a LLM host (or multiple hosts). - These hosts run API endpoints on their own PCs/servers for Horde users to access. The backends du jour are koboldcpp, a frontend for llama.cpp which is excellent, portable and literally one click, and KoboldAI, with the very fast and vram-efficient exllamav2 backend: https://github.com/LostRuins/koboldcpp https://github.com/henk717/KoboldAI - Hosts pick a quantized community LLM to run, which is (IMO) the real magic of this system. Cloud services tend to run generic Llama chat/instruct models,
Read the rest
OpenAI API models, or maybe a single proprietary finetune, but the Llama/Mistral finetuning community is red hot. New finetines and crazy merges/hybrids that outperform llama-chat in specific tasks (mostly Chat/Storytelling/RP) come out every day, and each one has a different "flavor" and format: https://huggingface.co/models?sort=modified&search=mistral+g... https://huggingface.co/models?sort=modified&search=13b+exl2 https://huggingface.co/models?sort=modified&search=20b - The horde workers then earn "kudos" for serving requests to clients. Anyone can use the kobold horde with no login, but when requests are queued, kudos earn hosts priority access to LLMs other hosts are running. I really like this scheme. Perverse incentives for spamming and such are minimal. Its free and easy for users on old hardware, but excessive leechers are deprioritized through the kudos system. It encourages experimentation with hosting (and trying) new finetunes, and gives hosts access to models they normally wouldn't know about or can't run. The horde API seems relatively simple and cheap to host, since its just text prompts/responses bouncing around. Its easy for me to host on a spare laptop, or in the background when I am not stressing my desktop GPU. There seems to be tons of interest in making LLMs "easy" to play with and giving access to GPU-poor users, but Kobold Horde has somehow flown under the radar.
In Chat with, and help host, a free community LLM "horde"
loudmax·Aug 28, 2023
You can absolutely run LLMs without a GPU, but you need to set expectations for performance. Some projects to look into are * llama.cpp - https://github.com/ggerganov/llama.cpp * KoboldCpp - https://github.com/LostRuins/koboldcpp * GPT4All - https://gpt4all.io/index.html llama.ccp will run LLMs that have been ported to the gguf format. If you have enough RAM, you can even run the big 70 billion parameter models. If you have a CUDA GPU, you can even offload part of the model onto the GPU and have the CPU do the rest, so you can get some partial performance benefit. The issue is that the big models run too slowly on a CPU to feel interactive. Without a GPU, you'll get much more reasonable performance running a smaller 7 billion parameter model instead. The responses won't be as good as the larger models, but they may still be go
Read the rest
od enough to be worthwhile. Also, development in this space is still coming extremely rapidly, especially for specialized models like ones tuned for coding.
In Continue with LocalAI: An alternative to GitHub's Copilot that runs locally
Aerroon·Aug 17, 2023
You could just try it out if you have the hardware at home. Grab KoboldCPP and a GGML model from TheBloke that would fit your RAM/VRAM and try it. Make sure you follow the prompt structure for the model that you will see on TheBloke's download page for the model (very important). KoboldCPP: https://github.com/LostRuins/koboldcpp TheBloke: https://huggingface.co/TheBloke I would start with a 13b or 7b model quantized to 4-bits just to get the hang of it. Some generic or story telling model. Just make sure you follow the prompt structure that the model card lists. KoboldCPP is very easy to use. You just drag the model file onto the executable, wait till it loads and go to the web interface.
In Why host your own LLM?
throwing_away·Aug 16, 2023
Sure thing. There's a bunch of ways to do it, but here's some quick notes on what I did. * arch linux has tons of `rocm` packages. I installed pretty much all of them: https://archlinux.org/packages/?sort=&q=rocm&maintainer=&fla... * you also need this one package from AUR: https://aur.archlinux.org/packages/opencl-amd * llama.cpp now has GPU support including "CLBlast", which is what we need for this, so compile with LLAMA_CLBLAST=ON * now you can run any model llama.cpp supports, so grab some ggml models that fit on the card from https://huggingface.co/TheBloke. * Test it out with: ./main -t 30 -ngl 128 -m huginnv1.2.ggmlv3.q6_K.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:" * You should see `BLAS = 1` in the llama.cpp output and you should get maybe 5 tokens p
Read the rest
er second on a 13b 6bit quantized ggml model. * You can compile llama-cpp-python with the same arguments and get text-generation-ui to work also, but there's a bit of dependency fighting to do it. * koboldcpp might be better, I just haven't tried it yet Hope that helps! Edit: just tried https://github.com/LostRuins/koboldcpp and it also works great. I should have started here probably. Compile with `make LLAMA_CLBLAST=1` run with ` python koboldcpp.py --useclblast 0 0 --gpulayers 128 huginnv1.2.ggmlv3.q6_K.bin`
In How Is LLaMa.cpp Possible?
brucethemoose2·Aug 16, 2023
Path of least resistance: - Download koboldcpp: https://github.com/LostRuins/koboldcpp - Download your 70B ggml model of choice, for instance airoboros 70B Q3_K_L: https://huggingface.co/models?sort=modified&search=70b+ggml - Run Koboldcpp with opencl (or rocm) with as many layers as you can manage on the GPU. If you use rocm, you need to install the rocm package from your linux distro (or direct from AMD on Windows). - Access the UI over http. Switch to instruct mode and copy in the correct prompt formatting from the model download page. - If you are feeling extra nice, get an AI Horde API key and contribute your idle time to the network, and try out other models on from other hosts: https://lite.koboldai.net/#
In How Is LLaMa.cpp Possible?
brucethemoose2·Aug 7, 2023
> Just ran Llama-2 (without a GPU) and it gave me coherent responses in 3 minutes (which is extremely fast for no GPU). How does this work? It should be much faster with llama.cpp. My old-ish laptop CPU (AMD 4900HS) can ingest a big prompt reasonably quickly and then stream text fast enough to (slowly) read. If you have any kind of dGPU, even a small laptop one, prompt ingestion is dramatically faster. Try the latest Kobold release: https://github.com/LostRuins/koboldcpp But to answer your question, the GGML CPU implementation is very good, and actually generating the response is somewhat serial, and more RAM speed bound than compute bound.
In Ask HN: How is Llama-2 optimized so much?
brucethemoose2·Jul 28, 2023
Try llama v1 13B with https://huggingface.co/izumi-lab/llama-13b-japanese-lora-v0-... There may be even larger llama tunes for Japanese... But huggingface is tricky to search. I would suggest baking the LORA into a ggml file and then running it with koboldcpp (with opencl offloading) for maximum ease of use. https://github.com/LostRuins/koboldcpp You could get even better results using Japanese llama.cpp grammar, but I have not been down that rabbit hole: https://github.com/ggerganov/llama.cpp/pull/1773
In Ask HN: Are there any LLMs specialised in translating Japanese to English?

Showing the 15 most recent of 19 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.