turbo-fieldfare

Why are we comparing to the maxed out model only? The M5 Ultra with 96 GB, 1 TB, and 64 GPU cores is $5499. Apple lets you lease that model for $110/month for 36 months. If you can settle for a M5 Max base model that would be a $49/month lease. Today, you should be able to run Qwen 3.8 - 27B amazingly well on either which is giving comparable performance to 5.6 Luna on SWE Bench. The local models are now getting better and more efficient and this should give you headroom. Tools like turbo fieldfare are really reducing the memory requirements to run large models and I don’t see it stopping soon. https://github.com/drumih/turbo-fieldfare

Why is anyone even using video cards these days? You may as well be burning cash. This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone: https://github.com/drumih/turbo-fieldfare https://github.com/leonickson1/Swiftlet https://github.com/sqliteai/warp

Yeah, original TurboFieldfare supports Gemma https://github.com/drumih/turbo-fieldfare

Thank you for testing and sharing results! I think I might understand your use case. You ssh the Mac and want something like `ollama run` with an interactive chat in terminal. Am I right? There is already experimental OpenAI-compatible server in this repo: ``` swift build -c release --product TurboFieldfareServer .build/release/TurboFieldfareServer \ --model scratch/gemma4.gturbo ``` After that a small terminal client can run inside the same ssh session and talk to `/v1/chat/completions` The client needs to keep a messages array, add each user message, send the full array with `stream:true`, print SSE chunks until `[DONE]`, then add the response back to the array. `/reset` can clear it There is a python example in the server docs. (https://github.com/drumih/turbo-fieldfare/blob/main/docs/OPE...) It is non-streaming, but can be use

Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are included. The trick is to keep the shared part of the model and the KV cache in RAM, then stream only the routed experts needed for each token from SSD. An SSD is way slower than RAM, so the runtime uses a small expert cache and bounded parallel `pread`. W