bitsandbytes

The problem is that we all have a lot of FUD (for good reasons). It's on AMD to solve that problem publically. They need to make it easier to understand what is supported so far and what's not. For example, for bitandbytes (a common dependency in LLM world) there's a ROCm fork that the AMD maintainers are trying to merge in (https://github.com/TimDettmers/bitsandbytes/issues/107). Meanwhile an Intel employee merged a change that made a common device abstraction (presumably usable by AMD + Apple + Intel etc.). There's a lot of that right now - super popular package that is CUDA-only is navigating how to make it work correctly with any other accelerator. We just need more information on what is supported.

Found it from https://github.com/Vahe1994/SpQR Was somehow expecting it to be at https://github.com/TimDettmers/bitsandbytes. My bad.

No. Without the inference code, the best we can have are guesses on its implementation, so the benchmark figures we can get could be quite wrong. It does seem better than Llama2-70B in my tests[0], which rely on the work done by Dmytro Dzhulgakov[1] and DiscoResearch[2]. But the point of releasing on bittorrent is to see the effervescence in hobbyist research and early attempts at MoE quantization, which are already ongoing[3]. They are benefitting from the community. [0]: https://twitter.com/espadrine/status/1733505332293288256 [1]: https://github.com/dzhulgakov/llama-mistral [2]: https://huggingface.co/DiscoResearch/mixtral-7b-8expert [3]: https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe

I have been researching methods and projects built around training and running LLMs locally. I'm interested in what others have been using on this front (including straight up Pytorch/Transformers). Here's what I've gathered so far: Engines/APIs: - vllm: Inference and serving engine for LLMs (none quantized models only?) [1] - ollama: Go project to run, create and share LLMs [2] - llama.cpp: Inference of LLaMA models in C/C++ w/UI (including quantized models) [3] - llama-cpp-python: run OpenAI compatible API bindings for llama.cpp [4] - llm-engine: engine for fine-tuning and serving LLMs [5] - Lamini: hosted? closed source? solution for training LLMs [6] - GPT4All: free, locally run chatbot [7] - SkyPilot: framework for running LLMs, AI, and batch jobs [8] - HuggingFace Transformers: APIs and tools to download and train

While on the surface, a 192GB Mac Studio seems like a great deal (it's not much more than a 48GB A6000!), there are several reasons why this might not be a good idea: * I assume most people have never used llama.cpp Metal w/ large models. It will drop to CPU speeds whenever the context window is full: https://github.com/ggerganov/llama.cpp/issues/1730#issuecomm... - while sure this might be fixed in the future, it's been an issue since Metal support was added, and is a significant problem if you are actually trying to actually use it for inferencing. With 192GB of memory, you could probably run larger models w/o quantization, but I've never seen anyone post benchmarks of their experiences. Note that at that point, the limited memory bandwidth will be a big factor. * If you are planning on using Apple Silicon for ML/training, I'd also be w

Release notes: https://github.com/TimDettmers/bitsandbytes/releases/tag/0.4...

Tim Dettmers is such a star. He's probably done more to make low-resource LLMs usable than anyone else. First bitsandbytes[1] and now this. [1] https://github.com/TimDettmers/bitsandbytes

The default loader doesn't seem to let you load quantized models but if you use something like https://github.com/oobabooga/text-generation-webui you can 1) use the model with `--load-in-8bit` which halves the memory (runs on my 24GB consumer card w/o an issue then, probably would fit on a 16GB card). There are also 4-bit quantized models and you can run probably `anon8231489123/vicuna-13b-GPTQ-4bit-128g --model_type LLaMA --wbits 4 --groupsize 128` although there have been reports that bitsandbytes have problems w/ 4bit perf on some cards: https://github.com/TimDettmers/bitsandbytes/issues/181