ds4

I started with antirez' DwarfStar[1] on one spark and that (~11-14tok/s generation, ~300-400 tok/s prompt processing) was enough of a taste for me to jump into 2 sparks, running the native quant of DSv4 Flash. Now at 40-50tok/s generation and ~2000 tok/s prefill with a model that I've seen reason through race conditions and be able to trivially pull off any straight-forward coding task, and remain coherent at 500k context. With a preview checkpoint of the weights! I'm excited for the future of local LLMs. There is some buy-in but apparently not an extreme amount to get access to models that can stand in the for the giants on all but the most challenging and/or hands-off coding tasks. [1]: https://github.com/antirez/ds4

indeed, there's even a (pretty solid) custom server just for DS4 https://github.com/antirez/ds4 -- works very well on high-RAM Macs

Claude-Code and Codex in combination, combined with an IDE such as Google Antigravity or VisualStudio-Code are very powerful tools, if your company can invest in hardware the new Mac Studio and MacBook Pro allow optimized local inference through open-source tools such as: https://github.com/antirez/ds4

ds4: prefill: 150.18 t/s, generation: 48.47 t/s together with https://github.com/antirez/ds4 + RTX Pro 6000, local provider :) I guess you could rent one too, get similar speed. https://gist.github.com/embedding-shapes/eceefa1b2699dca448f... The weights I've tried is the "DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix" ones which fits just about within 96GB VRAM. With some tuning, I've managed to get it to get up to ~60 t/s, I'm sure there is other things to do there too :)

> Trying to run them on a unified memory Mac > but still not quite in the realm of Sonnet or DeepSeek 4 Flash these are not mutually exclusive anymore. DS4 has set the bar for me these days. https://github.com/antirez/ds4

DwarfStar and other end-user inference engines should also support batched/concurrent inference IMHO. Not so much for the overly naïve "serving multiple users" case (the local hardware cannot really compete with ordinary datacenter gear, much less with the big proprietary suppliers; the compute headroom is too small to begin with once the model is in RAM) but rather to improve SSD streamed decode in the unattended inference scenario, where the goal is to meaningfully raise aggregate tok/s whilst facing an overly tight constraint on disk bandwidth, and CPU/GPU compute have a lot of slack. Of course this requires wide enough batches to have at least some reuse of fetched experts across a batch, but that seems feasible in the "unattended" case, where firing off multiple inferences to be processed together seems quite natural. (We may also ha

Since you have 512GB, might be worth looking into running deepseek4: https://github.com/antirez/ds4

If you have a 128GB Mac you really ought to try out: https://github.com/antirez/ds4 by the creator of redis. This is probably as close to it gets to state-of-the-art local LLM + agentic coding.

mbp16 m5 max 128gb, antirez/ds4, deepseekv4-flash. Works well for relatively dense (let's say <20k LoC per project) C codebases that are essentially a bunch of custom specialized stores, http servers, network infra, media transformers, etc. Runs through Pi with a custom prompt (basically "don't speculate blindly, isolate things, make them traceable and measurable, then verify") and behind a pretty restrictive bwrap setup - RO bind everything other than ~/.pi, cdw and a separate tmpfs, unshare almost everything other than the network - for which I use a network namespace that only allows tcp connections to a specific ip and port (i.e the inference mac) - i.e. netns exec into bwrap. Can't compare it to SOTA or higher-requirements models on what I work on - policy. That said, on a bunch of test pieces - it obviously isn't gpt-5.5, it definit

Waiting for this https://github.com/antirez/ds4 to stabilize for strix halo.

As an update, turns out Antirez created a brilliant 2 bit quant of Deepseek to fit into 128Gb systems along with a custom highlight tuned server. I've been running this the last few days and if I turn off envelope on OpenClaw, the performance is brilliant. Still to try with coding harnesses. It's a bit slow compared to the other models but so good that I'm willing to put up :-) https://github.com/antirez/ds4