club-3090

small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, and text inference on CUDA), and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA) This is local targeting high end consumer hardware like DGX Spark or AMD Ryzen AI Halo. For our mere mortals that were kids not long ago and can't really believe we've got our hands on a x090 series targeting Qwen3.8 27b, https://github.com/noonghunna/club-3090 is the way to go. I'm maintaining a web frontend for this, trying to at least. You can follow it here: https://github.com/gchamon/club-3090-server

With spec decode should easily get to >100tps on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090 make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446 devs about to get handed a two 5090 box each and told to max that out

you should look into https://github.com/noonghunna/club-3090

There's a sub 2k tier with a single 3090 that's also serviceable. Run https://github.com/noonghunna/club-3090 with beellama, fast inference at the cost of a reduced 102k context window

dense models are (more) compute heavy, so are generally worse to run on mac. mac tends to be better for (larger) MoE models. 27B dense can fit on a consumer graphics card. Even without getting into various "intrusive" ways to shrink the size of a model (e.g. REAP), something like a NVFP4 quant of Qwen3.6 27b https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4 should fit within ~22GB of VRAM. So easily on a 5090. It would also fit on a 3090/4090, but iirc they don't have NVFP4 natively, so you would want a different quant for them. you can see /r/LocalLLama for some discussions. See this (random) post about Qwen3.6-27B on a 3090 at ~100 tok/s https://www.reddit.com/r/LocalLLaMA/comments/1ujo46r/qwen_36... Note that it is possible you could still do this stuff with a mac, as there are ways of hooking up a eGPU to macs and using it for inference

I beg to differ. Have a look at this repo with single/double 3090 optimized configs for Qwen and Gema models: https://github.com/noonghunna/club-3090

This is a good place to start reading about dual gpus. https://github.com/noonghunna/club-3090/blob/master/docs/DUA...

If I started today, with building a server, I'd jump right into verified set-ups and writeups, like this one: https://github.com/noonghunna/club-3090 You can find info about running a patched version of vllm for 1x24gb, 2x and 4x. There's also quite a few "blackwell" subreddits, where people seem to share a lot of substantial information, if you're going the 6000 route.

This is not my experience at all. Even the Nous Research guys have stated that "Qwen3.6-27B is the canonical local model to use Hermes Agent with" [https://old.reddit.com/r/LocalLLaMA/comments/1sz2y76/ama_wit...]. I am finding the same when used with Pi and OpenCode. Gemma will just stop mid-tool call. It's been slower and I've had to reduce context size to run it. Qwen3.6 27b has been rock solid using club 3090's single card setup for agentic use -- https://github.com/noonghunna/club-3090/blob/master/docs/SIN...

I can see that and I don't know your setup, but there are people pushing >70t/s with MTP on a single 3090, with big contexts still >50t/s. 64k is not a lot for agentic coding, and IIRC 128k with turboquant and the likes should be possible for you. r/LocalLLM/ and r/LocalLLaMA/ are worth a visit IMO. EDIT: just found this recipe repo, may wanna give it a go: https://github.com/noonghunna/club-3090 EDIT-2: this can also shave off a lot of context need for tool calling -> https://github.com/rtk-ai/rtk

I run it with Llama.cpp on my RTX 3090. Also using the same Unsloth model. My config is similar to: https://github.com/noonghunna/club-3090/blob/master/docs/eng... I need to try out some of the other set ups mentioned in this repo for increased TPS.