GPU-Benchmarks-on-LLM-Inference

A 10GB 3080 still beats even an M2 Ultra with 192GB... memory bandwidth is not the only factor. https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

For LLM inference of batch size 1, it's hard to be saturate PCIe bandwidth specially for less powerful chips. You would get close to linear performance[1]. The obvious issue is few things on multiple GPU is harder, and many softwares don't fully support it or isn't optimized for it. [1]: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

That’s what I hope for, but everything that isn’t bananas expensive with unified memory has very low memory bandwidth. DGX (Digits), Framework Desktop, and non-Ultra Macs are all around 128 gb/s, and will produce single digits tokens per second for larger models: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen... So there’s a fundamental tradeoff between cost, inference speed, and hostable model size for the foreseeable future.

It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

> It's still "fast enough" for even 120b models in practice Is it? This is very subjective. The Mac Studio would not be "fast enough" for me on even a 70b model, not necessarily because its output is slow, but because the prompt evaluation speed is quite bad. See [0] for example numbers; on Llama 3 70B at Q4_K_M quantization, it takes an M2 Ultra with 192GB about 8.5 seconds just to evaluate a 1024-token prompt. A machine with 6 3090s (which would likely come in cheaper than the Mac Studio) is over 6 times faster at prompt parsing. A 120b model is likely going to be something like 1.5-2x slower at prompt evaluation, rendering it pretty much unusable (again, for me). [0] https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

LocalLLaMA subreddit usually has some interesting benchmarks and reports. Here is one example, testing performance of different GPUs and Macs with various flavours of Llama: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen... should give you consumer comparison