ai-benchmarks

Hmm, I'm not sure if I'm missing something but that 1st comment is what I'm doing. I have 3 different sized Deepseek R1 models (1.5, 8, 16) and they run on each board that can handle them and then the data is reported. For the 2nd, the file I grabbed initially was https://github.com/geerlingguy/ai-benchmarks/blob/main/obenc... - which I now notice wasn't modified in his repository, so I can check that out, but either way, the same version has been tested across everything thus far.

Wow, this repo and the ai-benchmarks repo are the ones I wanted https://github.com/geerlingguy/ai-benchmarks/issues/34 Thank you for doing these. Earned a star and a watch from me on both! Minor sponsor donation as gratitude. Would be sick to have an RSS feed for your data releases.

Decided to run an actual llama-bench run and let it go for the hour or two it needs. I'm posting my full results here (https://github.com/geerlingguy/ai-benchmarks/issues/47), but 8-10 t/s pp, and 7.99 t/s tg128, this is on a Pi 5 with no overclocking. Could probably increase the numbers slightly with an overclock. You need to have a fan/heatsink to get that speed of course, it's maxing out the CPU for the entire time.

See: https://github.com/geerlingguy/ai-benchmarks/issues/34

That and more in https://github.com/geerlingguy/ai-benchmarks/issues/34

Geerling benchmarked LLM performance on the Framework Desktop and the results look pretty lackluster to me. First, the software seems really immature. He couldn't get ROCm or the NPU working. When he finally got the iGPU working with Vulkan, he could only generate 5 tok/sec with Llama 3.1 70b (40 GB model). That's intolerably slow for anything interactive like coding or chatting imo, but I suppose that's a matter of opinion. https://github.com/geerlingguy/ollama-benchmark/issues/21

> If you're just doing AI stuff or GPU compute, this board might actually be a decent option, all things considered. Looking at the benchmark [0] When I first saw the board I thought this might be the RPi of the "AI age" since it is IMHO the most affordable option with 64Gb of RAM. But I am always cautious if we can really make the most of the so called "30 TOPS NPU" - [0] https://github.com/geerlingguy/ollama-benchmark/issues/13

Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...