SwiftLM

Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts): git clone --recursive https://github.com/SharpAI/SwiftLM.git cd SwiftLM swift build -c release # Trick to copy in that missing mlx.metallib file uv run --with mlx-metal python -c " import importlib.metadata, pathlib, shutil d = importlib.metadata.distribution('mlx-metal') metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib' shutil.copy(metallib, '.build/release/') print(f'Copied {metallib} -> .build/release/mlx.metallib') # Now start the server (downloads 69GB Qwen model) .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 But the server crashed when I tried to run a prompt through it: freed pointer was not the last allocation

git clone https://github.com/SharpAI/SwiftLM # no --recursive needed cd SwiftLM swift build -c release ### Please let me know if this fix the issue: # Copy metallib next to the binary (one-time step) cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To fit a 122B parameter model (e.g., Qwen3.5-122B MoE) without triggering macOS VM swapping or Watchdog kernel kills, the full ~60 GB weight file remains on NVMe. Only the top-k active expert pages are streamed to the GPU per forward pass at ~9 GB/s. As a result, inference runs with only 2,694 MB of active GPU VRAM on the M5 Pro 64GB, while the OS page cache automatically handles hot-expert reuse. By combining these two approache