mlx-lm

Yeah, first think I looked for on the post was MLX and it wasn't there. https://github.com/ml-explore/mlx-lm Having used half the systems that Vicki mentioned, mlx was the best balance between power and ease of use. Just a pip install away.

It's not implemented in mlx[1] yet (or llama.cpp[2]), so it may take a while. [1] https://github.com/ml-explore/mlx-lm/pull/990 [2] https://github.com/ggml-org/llama.cpp/pull/22673

Get into a venv, and run: > pip3 install git+https://github.com/ml-explore/mlx-lm.git > ./venv/bin/mlx_lm.generate --model "$MODEL" --temp 1.0 --top-p 0.95 --top-k 64 --max-tokens 128000 --prompt "Hello world" Where $MODEL is an unsloth model like: - unsloth/gemma-4-E4B-it-UD-MLX-4bit - unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit

> The same Gemma 4 MoE model (Q4) As you have so much RAM I would suggest running Q8_0 directly. It's not slower (perhaps except for the initial model load), and might even be faster, while being almost identical in quality to the original model. And just to be sure: you're are running the MLX version, right? The mlx-community quantization seemed to be broken when I tried it last week (it spit out garbage), so I downloaded the unsloth version instead. That too was broken in mlx-lm (it crashed), but has since been fixed on the main branch of https://github.com/ml-explore/mlx-lm. I unfortunately only have 16 GiB of RAM on a Macbook M1, but I just tried to run the Q8_0 GGUF version on a 2023 AMD Framework 13 with 64 GiB RAM just using the CPU, and that works surprisingly well with tokens/s much faster than I can read the output. The prompt c

benchmarks: we provide plenty in the over 100 page tech report here https://github.com/swiss-ai/apertus-tech-report/blob/main/Ap... quantizations: available now in MLX https://github.com/ml-explore/mlx-lm (gguf coming soon, not trivial due to new architecture) model sizes: still many good dense models today lie in the range between our small and large chosen sizes

Hey Simon, VB from Hugging Face here and the person who added the model to MLX and llama.cpp (with Son). The PR hasn't yet landed on llama.cpp, hence it doesn't work OTB on llama.cpp installed via brew (similarly doesn't work with ollama since they need to bump their llama.cpp runtime) The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp If you want to avoid it, I added SmolLM3 to MLX-LM as well: You can run it via `mlx_lm.chat --model "mlx-community/SmolLM3-3B-bf16"` (requires the latest mlx-lm to be installed) here's the MLX-lm PR if you're interested: https://github.com/ml-explore/mlx-lm/pull/272 similarly, llama.cpp here: https://github.com/ggml-org/llama.cpp/pull/14581 Let me know if you face any issues!

For those who never heard of those: mlx is similar to numpy/pytorch, but only for Apple Silicon. mlx-lm is a llama.cpp equivalent, but built on top of mlx. https://github.com/ml-explore/mlx-lm

I've shared a bunch of notes on MLX over the past year, many of them with snippets of code I've used to try out models: https://simonwillison.net/tags/mlx/ I mainly use MLX for LLMs (with https://github.com/ml-explore/mlx-lm and my own https://github.com/simonw/llm-mlx which wraps that), vision LLMs (via https://github.com/Blaizzy/mlx-vlm) and running Whisper (https://github.com/ml-explore/mlx-examples/tree/main/whisper) I haven't tried mlx-audio yet (which can synthesize speech) but it looks interesting too: https://github.com/Blaizzy/mlx-audio The two best people to follow for MLX stuff are Apple's Awni Hannun - https://twitter.com/awnihannun and https://github.com/awni - and community member Prince Canuma who's responsible for both mlx-vlm and mlx-audio: https://twitter.com/Prince_Canuma and https://github.com/Blaizzy