omlx

I've been working with local models for the past year. There's so many possibilities, but I don't think coding is one. Coding requires so many layers beyond inference; I spent so much time trying to replicate what Claude Code does end to end locally. Understanding all the layers and keeping up with the advancements feels like a slog. Even this article messes up and misunderstands what some of the settings are doing. Qwen in particular seems to work at first, then often gets stuck in thought loops when used for actual work. However, text-to-speech, speech-to-text, and non-code LLM use cases are so useful to have local, and don't require big hardware. Having a universal reliable inference engine interface, I think, is the big unlock that needs to happen before app devs can ship these features. Personal concrete use case: meeting recording a

oMLX (https://github.com/jundot/omlx) makes running the mlx inference server quite easy for those interested in UI-based hosting. oMLX also supports mtp or dflash drafting.

https://github.com/jundot/omlx note: 27b is going to be slow; use the 35b MoE if you want decent token/sec speed.

Try https://github.com/jundot/omlx I've been using qwen3.5 (122b) with claude code for months, and it's definitely dumber than sonnet/opus, but it works through things reasonably well (i.e. writes half-decent code and tool calls usually work), and I pretty much never run into loops now. and make sure you're following Unsloth's recommendations for temperature/etc.

Just get a decent macbook, use LM Studio or OMLX and the latest qwen model you can fit in unified ram. Hooking up Claude Code to it is trivial with omlx. https://github.com/jundot/omlx

how about the others: - vLLM https://vllm.ai/ ? - oMLX https://github.com/jundot/omlx ?

I run mlx models with omlx[1] on my mac and it works really well. [1] https://github.com/jundot/omlx

Hi HN, I'm the developer. I built oMLX because every Mac LLM server I tried fell apart with coding agents. The core problem: tools like Claude Code and OpenClaw send requests where the prompt prefix keeps shifting. Most servers invalidate the KV cache after each turn, forcing a full re-prefill. On large contexts (50-100K tokens), that means 30-90 seconds of waiting per request. After a few turns, it's practically unusable. oMLX solves this with paged SSD caching. Every KV cache block is persisted to disk. When a previous prefix comes back, it's restored from SSD instead of being recomputed. In practice, TTFT drops from 30-90s to 1-3s on follow-up requests. Cache blocks survive server restarts too. What's under the hood: - Continuous batching via mlx-lm (multiple concurrent requests) - Multi-model serving (LLM + VLM + Embedding + Reranker

I was frustrated that coding agents like Claude Code were basically unusable with local models. every few turns the prefix shifts, KV cache gets invalidated, and your mac has to re-prefill the entire context from scratch. So i built oMLX. it persists KV cache blocks to SSD, and when a previous context comes back, it restores from disk instead of recomputing. this alone made Qwen3-Coder-80B on my M3 Ultra actually usable for real coding sessions. Some other stuff it does: continuous batching, multi-model serving (LLM + embedding + reranker at once), prefix sharing with copy-on-write, and a native mac menubar app so you don't have to touch the terminal. Just shipped a built-in benchmark tool too, so you can test your own setup easily.

I built an open-source LLM inference server optimized for Apple Silicon. The main motivation was coding agents - tools like Claude Code send requests where the context prefix keeps shifting, invalidating KV cache. A few turns later the agent circles back, and your Mac has to re-prefill the entire context from scratch. oMLX solves this with paged SSD caching. Every KV cache block is persisted to disk. When a previous prefix returns, it's restored instantly instead of being recomputed. This makes long coding sessions significantly faster. It also supports continuous batching for concurrent requests, multi-model serving (LLM + embedding + reranker) with LRU eviction, block-level KV cache with prefix sharing and copy-on-write, OpenAI and Anthropic compatible APIs, and tool calling. Ships as a signed macOS menubar app with a web dashboard. Git