cactus

AM with the OS, you get thermally throttled, and the same model behaves differently on different hardware. So we wrote a runtime from scratch for resource-constrained devices. Since then, Cactus has grown to process millions of weekly inference runs, and tens of thousands of monthly active developers. Our biggest learning from deploying Cactus in production apps is that while local models can handle 90% of workloads, that 10% gap means they're still not production-ready. Our users' fix was to build custom cloud fallback logic. Cactus v2 fixes that: Our approach to cloud fallback is to post-train a probe into the model's weights that reads its internal activations and emits a confidence signal. This way, the routing happens inside the model rather than in a prompt classifier sitting in front of it. We believe this is critical for multi-turn agentic work, where the model should know which turns are easy enough to be handled locally, and which are hard - and get handed off to the cloud. What ships today is single-turn routing for Gemma-4 E2B against a configurable escalation endpoint (Gemini, Claude, OpenAI-compatible, or your own endpoint). Our next target is hybrid-native models for multi-turn agentic work. This is the genuinely unsolved problem. Our current probe-in-the-weights is showing promising results and we look to release the first model variants soon. The hybrid variants are Gemma derivatives, released under the Gemma terms noted on the HF cards. In addition to the hybrid routing, the runtime has SOTA quantization, which is lossless at 4bit, memory maps weights to decrease RAM footprint and runs cross-platform, with Python, Rust, React Native, Swift, and Kotlin bindings. Disclaimer: Cactus is distributed as source-available - free for personal use and small companies; commercial license above that (Docker-style license). You can get started on our GitHub: https://github.com/cactus-compute/cactus or by `brew install cactus-compute/cactus/cactus`.

watches, glasses...). Training: - Pretrained on 200B tokens across 16 TPU v6e (27 hours) - Post-trained on 2B tokens of synthesized function-calling data (45 minutes) - Dataset synthesized via Gemini with 15 tool categories (timers, messaging, navigation, smart home, etc.) You can test it right now and finetune on your Mac/PC: https://github.com/cactus-compute/needle The full writeup on the architecture is here: https://github.com/cactus-compute/needle/blob/main/docs/simp... We found that the "no FFN" finding generalizes beyond function calling to any task where the model has access to external structured knowledge (RAG, tool use, retrieval-augmented generation). The model doesn't need to memorize facts in FFN weights if the facts are provided in the input. Experimental results to published. While it beats FunctionGemma-270M, Qwen-0.6B, Granite-350M, LFM2.5-350M on single-shot function calling, those models have more scope/capacity and excel in conversational settings. We encourage you to test on your own tools via the playground and finetune accordingly. This is part of our broader work on Cactus (https://github.com/cactus-compute/cactus), an inference engine built from scratch for mobile, wearables and custom hardware. We wrote about Cactus here previously: https://news.ycombinator.com/item?id=44524544 Everything is MIT licensed. Weights: https://huggingface.co/Cactus-Compute/needle GitHub: https://github.com/cactus-compute/needle

es but 70% of phones today are low-mid budget. - Bloated app bundle sizes and battery drain are serious concerns for users. - Phone GPU battery drain is unacceptable, NPUs are preferred, but few phones have those for now. - Some are platform-specific, requiring different models and workflows for different operating systems. At Cactus, we’ve written kernels and inference engine for running AI locally on any phone, from the ground-up. Cactus is designed for mobile devices and their constraints. Every design choice like energy efficiency, accelerator support, quantization levels, supported models, weight format, and context management were determined by this. We also provide minimalist SDKs for app developers to build agentic workflows in 2-5 lines of code. We made a Show HN post when we started the project to get the community's thoughts (https://news.ycombinator.com/item?id=44524544). Based on your feedback, we built Cactus bottom-up to solve those problems, and are launching the Cactus Kernels, Cactus Graph and Cactus Engine, all designed for phones and tiny devices. CPU benchmarks for Qwen3-600m-INT8 : - 16-20 toks/sec on Pixel 6a / Galaxy S21 / iPhone 11 Pro - 50-70 toks/sec on Pixel 9 / Galaxy S25 / iPhone 16. - Time-to-first-token is as low as 50ms depending on prompt size. On NPUs, we see Qwen3-4B-INT4 run at 21 toks/sec. We are open-source (https://github.com/cactus-compute/cactus). Cactus is free for hobbyists and personal projects, with a paid license required for commercial use. We have a demo app on the App Store at https://apps.apple.com/gb/app/cactus-chat/id6744444212 and on Google Play at https://play.google.com/store/apps/details?id=com.rshemetsub.... In addition, there are numerous apps using Cactus in production, including AnythingLLM (https://anythingllm.com/mobile) and KinAI (https://mykin.ai/). Collectively they run over 500k weekly inference tasks in production. While Cactus can be used for all Apple devices including Macbooks due to their design, for computers/AMD/Intel/Nvidia generally, please use HuggingFace, Llama.cpp, Ollama, vLLM, MLX. They're built for those, support x86, and are all great! Thanks again, please share your thoughts, we’re keen to understand your views.

are built with these today. - Supports any GGUF model you can find on Huggingface; Qwen, Gemma, Llama, DeepSeek, Phi, Mistral, SmolLM, SmolVLM, InternVLM, Jan Nano etc. - Accommodates from FP32 to as low as 2-bit quantized models, for better efficiency and less device strain. - Have MCP tool-calls to make them performant, truly helpful (set reminder, gallery search, reply messages) and more. - Fallback to big cloud models for complex, constrained or large-context tasks, ensuring robustness and high availability. It's completely open source. Would love to have more people try it out and tell us how to make it great! Repo: https://github.com/cactus-compute/cactus