ninfer

This is interesting, thanks. - https://github.com/Neroued/ninfer

Just a couple days ago I learned about ninfer (https://github.com/Neroued/ninfer) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.

try ninfer once you get your 5090 https://github.com/Neroued/ninfer

Good 60-70 tg and 2K pp Qwen 3.8 27B FP8 can be had for about 5-6K (2xR9700 + PC) Gives about 3-4 concurrent sessions with full 262K Fast 150+ tg and 2-8K pp Qwen 3.8 27B nvfp4 is about 8K (5090 +PC) Gives really only one concurrent session that flies because kv caching is not perfect for ninfer https://github.com/Neroued/ninfer both are very serviceable, I prefer FP8 on 2xR9700 But, yes it doesn't scale that well but in 5 years the same hardware should still be very capable of running some great MoE models, for example Qwen 3.6 35BA3B on 5090 can fly at 600 tg

Strongly recommend https://github.com/Neroued/ninfer, which can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B-A3B.

Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS. Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow. The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking. However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not

Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures. https://github.com/Neroued/ninfer/blob/master/docs/performan... Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural chec

If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).

If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.