koboldcpp

Rolling build of Koboldcpp now supports Krea2: * https://github.com/LostRuins/koboldcpp/releases/tag/rolling Model weights: * https://github.com/leejet/stable-diffusion.cpp/blob/master/d... In the launcher GUI, select 'Image Gen' and set Krea2 Turbo gguf as the image model, Qwen3 VL 4b gguf as the T5 model, and Wan2.1 safetensors as the VAE.

Koboldcpp supports image generation, but you will have to wait for the next release for Krea2 support. * https://github.com/LostRuins/koboldcpp

Koboldcpp has built in support for image models. Model search and download, one executable to run, UI, OpenAI API endpoint, llama.cpp endpoint, highly configurable. If you want to get up and running instantly, just pick a kcppt file and open that and it will download everything you need and load it for you. Engine: * https://github.com/LostRuins/koboldcpp/releases/latest/ Kcppt files: * https://huggingface.co/koboldcpp/kcppt/tree/main

Try koboldcpp with the kcppt config file. The easiest way by far. Download the release here * https://github.com/LostRuins/koboldcpp/releases/tag/v1.103 Download the config file here * https://huggingface.co/koboldcpp/kcppt/resolve/main/z-image-... Set +x to the koboldcpp executable and launch it, select 'Load config' and point at the config file, then hit 'launch'. Wait until the model weights are downloaded and launched, then open a browser and go to: * http://localhost:5001/sdui EDIT: This will work for Linux, Windows and Mac

Oh, that horrible Electron UI. Under Windows it pegs a core on my CPU at all times! If you're just working as a single user via the OpenAI protocol, you might want to consider koboldcpp. It bundles a GUI launcher, then starts in text-only mode. You can also tell it to just run a saved configuration, bypassing the GUI; I've successfully run it as a system service on Windows using nssm. https://github.com/LostRuins/koboldcpp/releases Though there are a lot of roleplay-centric gimmicks in its feature set, its context-shifting feature is singular. It caches the intermediate state used by your last query, extending it to build the next one. As a result you save on generation time with large contexts, and also any conversation that has been pushed out of the context window still indirectly influences the current exchange.

It would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#do... docker image was broken for me on 7800xt running rhel9 https://github.com/Atinoda/text-generation-webui-docker

MiniCPM-V 2.6 is based on Qwen 2 and is also great at handwriting. It works locally with KoboldCPP. Here are the results I got with a test I just did. Image: * https://imgur.com/wg0kdQK Output: * https://pastebin.com/RKvYQasi OCR script used: * https://github.com/jabberjabberjabber/LLMOCR/blob/main/llmoc... Model weights: MiniCPM-V-2_6-Q6_K_L.gguf, mmproj-MiniCPM-V-2_6-f16.gguf Inference: * https://github.com/LostRuins/koboldcpp/releases/tag/v1.75.2

Download koboldcpp and llama3.1 gguf weights, use it with the llama3 completions adapter. Edit the 'background.js' file in the extension and replace the openAI endpoint with 'http://your.local.ip.addr:5001/v1/chat/completions' Set anything you want as an API key. Now you have a truly local version. * https://github.com/LostRuins/koboldcpp/releases * https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-... * https://github.com/LostRuins/koboldcpp/blob/concedo/kcpp_ada...

I believe if you search around you might be able to some blockchain that uses that technology with AI to run models. I just did a quick search and found this[1]. Also there's AI Horde[2] that uses the KoboldAI[3] project for distributed compute. 1 - https://www.deepbrainchain.org/ 2 - https://horde.koboldai.net/ 3 - https://github.com/LostRuins/koboldcpp

Llava just integrates Clip with a llama model. Koboldcpp can now do this with many models out of the box: * https://github.com/LostRuins/koboldcpp/releases/tag/v1.61.2

Nice, adding these to my list. Here's a list that I put together, it has active GitHub projects for LLM UIs, ordered by stars: - https://github.com/nomic-ai/gpt4all - https://github.com/imartinez/privateGPT - https://github.com/oobabooga/text-generation-webui - https://github.com/FlowiseAI/Flowise - https://github.com/lobehub/lobe-chat - https://github.com/PromtEngineer/localGPT - https://github.com/h2oai/h2ogpt - https://github.com/huggingface/chat-ui - https://github.com/SillyTavern/SillyTavern - https://github.com/ollama-webui/ollama-webui - https://github.com/Chainlit/chainlit - https://github.com/LostRuins/koboldcpp - https://github.com/ParisNeo/lollms-webui/

Sadly I can't try this because I'm on Windows or Linux. Was testing apps like this if anyone is interested: Best / Easy to use: - https://lmstudio.ai - https://msty.app - https://jan.ai More complex / Unpolished UI: - https://gpt4all.io - https://pinokio.computer - https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generat... - https://github.com/LostRuins/koboldcpp Misc: - https://faraday.dev (AI Characters): No UI / Command line (not for me): - https://ollama.com - https://privategpt.dev - https://serge.chat - https://github.com/Mozilla-Ocho/llamafile Pending to check: - https://recurse.chat Feel free to recommend more!