ggllm.cpp

It doesn't support Falcon right now, but there's a fork that does (https://github.com/cmp-nct/ggllm.cpp/).

Pretty much anything with 32GB (?) total RAM+VRAM: https://github.com/cmp-nct/ggllm.cpp But its going to be slow without even a small Nvidia GPU (a 2060?). CPUs are really slow at prompt ingestion, and that can't be hidden with streaming.

The GGLLM fork seems to be the leading falcon winner for now [1] It comes with its own variant of the GGML sub format "ggcv1" but there's quants available on HF [2] Although if you have a GPU I'd go with the newly released AWQ quantization instead [3] the performance is better. (I may or may not have a mild local LLM addiction - and video cards cost more then drugs) [1] https://github.com/cmp-nct/ggllm.cpp [2] https://huggingface.co/TheBloke/falcon-7b-instruct-GGML [3] https://huggingface.co/abhinavkulkarni/tiiuae-falcon-7b-inst...

For base tooling, things like: https://huggingface.co/ (finding models and downloading them) https://github.com/ggerganov/llama.cpp (llama) https://github.com/cmp-nct/ggllm.cpp (falcon) For interactive work (art/chat/research/playing around), things like: https://github.com/oobabooga/text-generation-webui/blob/main... (llama) (Also - they just added a decent chat server built into llama.cpp the project) https://github.com/invoke-ai/InvokeAI (stable-diffusion) Plus a bunch of hacked together scripts. Some example models (I'm linking to quantized versions that someone else has made, but the tooling is in the above repos to create them from the published fp16 models) https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... etc.

Experimental Falcon inference via ggml (so on CPU): https://github.com/cmp-nct/ggllm.cpp It has problems but it does work