open-jet

Qwen3.8-27b can be ran at 100 tok/sec average https://github.com/L-Forster/open-jet/tree/master/megakernel. Gemma4:31b, which is the better model with more complex architecture (and thus can't be compiled into a megakernel), is gonna be way slower in prefill, and ~50% slower than a 3090 even with oMLX. You can't get away from much slower shared ram vs vram, and less parallelism compared to 3090. The other advantage of an eGPU setup, you can simply add/swap performance. Dual 4070s give you 24b of ram and you can still get about 80% of the performance of a single 3090, and actually you get faster prefill due to more parallelism. So in terms of money/usefulness, Mac always loses.

>Qwen 3.8 27B Q4 with 64k, 3090: ~40tok/sec Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel. And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_... Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec. And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time. Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for runn

Hi HN, I am building a terminal UI for self-hosted AI agents on Jetsons and other edge devices with unified memory. The reason I started it was that most local agent harnesses seems aimed at machines with plenty of RAM and a stable internet-connected developer environment. On Jetson-class hardware, the annoying problems are different: context growth eats memory, sessions break, models may fit but leave very little headroom, and a lot of tools assumes cloud access. Recent additions include: - air-gapped mode - automatic context condensing under memory pressure - persistent memory files and /memory controls - harness modes for chat/code/review/debug workflows - replayable traces for evals/debugging - multimodal local image input - OpenTelemetry support I’d love for you to try it out. The code is up on GitHub, and contributions/roasts

Hi HN, I am building a Terminal User Interface (like Claude Code) for self-hosted AI agents on Jetsons. Works in air-gapped environments. Unlike other solutions, this is optimised for unified memory machines, as to avoid OOM errors. The agent can do stuff like edit, read, create files - manage and interpret data locally. Currently, it gets ~17 tok/s on Jetson Orin Nano 8GB using Qwen3-4B-Instruct-4bit In the future, adding TensorRT .engine support which will boost inference further. I am trying to get the memory footprint down, so if anyone has knowledge on kv cache optimisation, that would be great. I would love to get your feedback and people try running it on more capable devices and models - post your results here. Run ``` pip install open-jet open-jet --setup ``` Webiste: https://www.openjet.dev/ Directly on Pypi: https://pypi.org/pr