koboldcpp

Try koboldcpp with the kcppt config file. The easiest way by far. Download the release here * https://github.com/LostRuins/koboldcpp/releases/tag/v1.103 Download the config file here * https://huggingface.co/koboldcpp/kcppt/resolve/main/z-image-... Set +x to the koboldcpp executable and launch it, select 'Load config' and point at the config file, then hit 'launch'. Wait until the model weights are downloaded and launched, then open a browser and go to: * http://localhost:5001/sdui EDIT: This will work for Linux, Windows and Mac

Oh, that horrible Electron UI. Under Windows it pegs a core on my CPU at all times! If you're just working as a single user via the OpenAI protocol, you might want to consider koboldcpp. It bundles a GUI launcher, then starts in text-only mode. You can also tell it to just run a saved configuration, bypassing the GUI; I've successfully run it as a system service on Windows using nssm. https://github.com/LostRuins/koboldcpp/releases Though there are a lot of roleplay-centric gimmicks in its feature set, its context-shifting feature is singular. It caches the intermediate state used by your last query, extending it to build the next one. As a result you save on generation time with large contexts, and also any conversation that has been pushed out of the context window still indirectly influences the current exchange.