cactus

AM with the OS, you get thermally throttled, and the same model behaves differently on different hardware. So we wrote a runtime from scratch for resource-constrained devices. Since then, Cactus has grown to process millions of weekly inference runs, and tens of thousands of monthly active developers. Our biggest learning from deploying Cactus in production apps is that while local models can handle 90% of workloads, that 10% gap means they're still not production-ready. Our users' fix was to build custom cloud fallback logic. Cactus v2 fixes that: Our approach to cloud fallback is to post-train a probe into the model's weights that reads its internal activations and emits a confidence signal. This way, the routing happens inside the model rather than in a prompt classifier sitting in front of it. We believe this is critical for multi-turn agentic work, where the model should know which turns are easy enough to be handled locally, and which are hard - and get handed off to the cloud. What ships today is single-turn routing for Gemma-4 E2B against a configurable escalation endpoint (Gemini, Claude, OpenAI-compatible, or your own endpoint). Our next target is hybrid-native models for multi-turn agentic work. This is the genuinely unsolved problem. Our current probe-in-the-weights is showing promising results and we look to release the first model variants soon. The hybrid variants are Gemma derivatives, released under the Gemma terms noted on the HF cards. In addition to the hybrid routing, the runtime has SOTA quantization, which is lossless at 4bit, memory maps weights to decrease RAM footprint and runs cross-platform, with Python, Rust, React Native, Swift, and Kotlin bindings. Disclaimer: Cactus is distributed as source-available - free for personal use and small companies; commercial license above that (Docker-style license). You can get started on our GitHub: https://github.com/cactus-compute/cactus or by `brew install cactus-compute/cactus/cactus`.