koboldcpp

It would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#do... docker image was broken for me on 7800xt running rhel9 https://github.com/Atinoda/text-generation-webui-docker

MiniCPM-V 2.6 is based on Qwen 2 and is also great at handwriting. It works locally with KoboldCPP. Here are the results I got with a test I just did. Image: * https://imgur.com/wg0kdQK Output: * https://pastebin.com/RKvYQasi OCR script used: * https://github.com/jabberjabberjabber/LLMOCR/blob/main/llmoc... Model weights: MiniCPM-V-2_6-Q6_K_L.gguf, mmproj-MiniCPM-V-2_6-f16.gguf Inference: * https://github.com/LostRuins/koboldcpp/releases/tag/v1.75.2

Download koboldcpp and llama3.1 gguf weights, use it with the llama3 completions adapter. Edit the 'background.js' file in the extension and replace the openAI endpoint with 'http://your.local.ip.addr:5001/v1/chat/completions' Set anything you want as an API key. Now you have a truly local version. * https://github.com/LostRuins/koboldcpp/releases * https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-... * https://github.com/LostRuins/koboldcpp/blob/concedo/kcpp_ada...

I believe if you search around you might be able to some blockchain that uses that technology with AI to run models. I just did a quick search and found this[1]. Also there's AI Horde[2] that uses the KoboldAI[3] project for distributed compute. 1 - https://www.deepbrainchain.org/ 2 - https://horde.koboldai.net/ 3 - https://github.com/LostRuins/koboldcpp

Llava just integrates Clip with a llama model. Koboldcpp can now do this with many models out of the box: * https://github.com/LostRuins/koboldcpp/releases/tag/v1.61.2

Nice, adding these to my list. Here's a list that I put together, it has active GitHub projects for LLM UIs, ordered by stars: - https://github.com/nomic-ai/gpt4all - https://github.com/imartinez/privateGPT - https://github.com/oobabooga/text-generation-webui - https://github.com/FlowiseAI/Flowise - https://github.com/lobehub/lobe-chat - https://github.com/PromtEngineer/localGPT - https://github.com/h2oai/h2ogpt - https://github.com/huggingface/chat-ui - https://github.com/SillyTavern/SillyTavern - https://github.com/ollama-webui/ollama-webui - https://github.com/Chainlit/chainlit - https://github.com/LostRuins/koboldcpp - https://github.com/ParisNeo/lollms-webui/

Sadly I can't try this because I'm on Windows or Linux. Was testing apps like this if anyone is interested: Best / Easy to use: - https://lmstudio.ai - https://msty.app - https://jan.ai More complex / Unpolished UI: - https://gpt4all.io - https://pinokio.computer - https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generat... - https://github.com/LostRuins/koboldcpp Misc: - https://faraday.dev (AI Characters): No UI / Command line (not for me): - https://ollama.com - https://privategpt.dev - https://serge.chat - https://github.com/Mozilla-Ocho/llamafile Pending to check: - https://recurse.chat Feel free to recommend more!