cua

s: what happens if you give a model an interface of current context, and a set of possible choices, and you ask it to return a probability for each choice? This kind of model does not generate output token by token like most LLMs do, but rather scores the options you give it, which you can check, trust, and use to drive your app's behavior. CUA-S1 is our answer for narrow, specialized decision models for computer use. Our first release is CUA-S1-FORMS. We built this from ideas and code in jevlike, and then trained a second model just to handle form interactions. It has 706k parameters, and the original checkpoint is 2.8 MB. The first training iteration took less than 30 minutes on synthetic data. Given a set of structured elements and values extracted from a document, it predicts whether to use the given value, CHECK, CLICK, or SKIP for each element. It does not predict new values for text fields, and does not consider screenshots. Element decisions are scored together, and your code can order the actions, and Cua Driver will execute them one at a time. A first evaluation of this specialist vs. hosted Jev on our form task: - For the whole decision set: 99.7% correct vs 83.6%. - For the subset of steps that require an action: 100% correct vs 96%. - For the subset of steps that are just leaving already-filled fields alone: 100% correct vs 74%. The specialist was trained specifically for this task and convention (just press skip for already filled boxes), while hosted Jev has not been fine-tuned for it, so this is an experiment in scoped specialization. We measured 7-9 ms to score a form locally vs. 260-280 ms per call to hosted Jev including network latency, though those samples measure different things and are not end-to-end form completion times. Our interest here is in the space between a brittle script and a general agent loop. The content and layout of form fields vary enough that scripts get unwieldy, but the set of available decisions can remain narrow and well scoped. We want to explore the possibility of a general agent encountering something novel, and passing well understood decisions over to specialists like this. That is a direction we are looking into. The current release is for forms only. We're open sourced the synthetic data generation, training, evaluation, and Driver integration under libs/cua-s1 with an MIT license. Comments welcome! Especially if you are building computer-use agents and have run into a recurring decision that is too variable to script but is too narrow to call another LLM for.

not at all by the restrictions of the FSL. The only entities this really restricts are F/OSS distributions which might otherwise include Tart as a first-class package in their distros. :-\ -- 1: https://web.archive.org/web/20260412071019/https://cirruslab... 2: https://github.com/trycua/cua/tree/main/libs/lume

ons with the human or other agents using the same machine. We realized macOS has no first-class API for "drive this app without touching the cursor". CGEventPost routes through the hardware input stream, so it moves your cursor. CGEvent.postToPid avoids the cursor warp, but Chromium treats those events as untrusted and silently drops clicks at the renderer boundary. Activating the target app first raises the window and pulls focus, defeating the point of background execution. Cua Driver is our attempt at a real fix: a background computer-use driver for macOS that lets an agent click, type, scroll, and read native apps while your cursor, frontmost app, and Space stay where they are. The default interface is a CLI, so it is easy to script or call from any coding agent shell. Try it on macOS 14+: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-d...)" The first internal use case was delegated demo recording. We ask Claude Code to drive an app while 'cua-driver recording start' captures the trajectory, screenshots, actions, and click markers. The result is an agent-generated product demo, Screen Studio inspired. Other things we have used it for: - Replacing Vercel’s agent-browser and other browser-use CLIs. With Claude Code and Cua Driver, you do not need Chrome DevTools Protocol at all. - A dev-loop QA agent that reproduces a visual bug, edits code, rebuilds, and verifies the UI while my editor stays frontmost. - Personal-assistant flows that use iMessage from Claude Code, Hermes, or other general-purpose agent CLIs. - Pulling visual context from Chrome, Figma, Preview, or YouTube windows I am not looking at, without relying on their APIs. What made this harder than expected: - CGEventPost warps the cursor because it goes through the HID stream. - CGEvent.postToPid does not warp the cursor, but Chromium drops it at the renderer IPC boundary. - Activating the target first raises the window and can drag you across Spaces. - Electron apps stop keeping useful AX trees alive when windows are occluded without a private remote-aware SPI. The unlock was SkyLight. SLEventPostToPid is a sibling of the public per-PID call, but it travels through a WindowServer channel Chromium accepts as trusted. Pair it with yabai’s focus-without-raise pattern, plus an off-screen primer click at (-1, -1), and the click lands without the window ever raising. One thing we learned: the right addressing mode depends on the app. Native macOS apps usually have rich AX trees, Chromium-family apps often need a hybrid of AX and screenshots, and apps like Blender or CAD tools may expose almost no useful AX surface. The mistake is defaulting to pixels everywhere - or defaulting to AX everywhere. Long technical writeup: https://github.com/trycua/cua/blob/main/blog/inside-macos-wi... I would like feedback from people building Mac automation, agent harnesses, or accessibility tooling. If it breaks on an macOS app you care about, that is useful data for us.

-use or https://github.com/MrAliHasan/Sophia-AI-Assistant https://github.com/TurixAI/TuriX-CUA https://github.com/iBz-04/raya https://github.com/coasty-ai/open-computer-use https://github.com/OthersideAI/self-operating-computer... I mean there's dozens.

ll desktop isolated. # Run agents in parallel: $ npx cuabot -n research openclaw $ npx cuabot -n coding codex # Or script the CLI: $ npx cuabot libreoffice --writer & $ npx cuabot --click 150 48 $ npx cuabot --type “I Cua!” Right now my cuabot agent is exploring mobile/desktop apps to turn into cuabench RL environments. I watch the windows appear, intervene when it gets stuck, and let it continue until it opens the completed GUI gym for me to interact with. Why we built this: We built Cua (https://github.com/trycua/cua), our open-source SDK for building and benchmarking computer-use systems with GUI sandboxes. We kept seeing two common patterns: 1. Agent screenshots your desktop and controls your mouse – Works with your data, but unsafe and locks you out 2. Agent runs in a sandbox with an external VNC desktop – Safer (our default), but clunky to monitor, hard to interact with, and tedious for data transfer General computer-use should be frictionless. Asking your agent to debug a GUI app shouldn't require opening an entire desktop stream. The GUI app should just appear alongside your windows, sandboxed and ready. How it works: cuabot [command] launches cuabotd, which manages a Ubuntu + Xpra Docker container, a multi-cursor overlay, an Xpra computer-use MCP server, and an Xpra seamless client. It auto-configures your agent (Claude, Aider, etc.) to connect to the computer-use MCP, then pipes terminal I/O through WebSocket. The Xpra client automatically detects and streams windows launched in the container, with H.264 encoding, audio, and customizable clipboard sharing. Since the computer-use MCP interacts through an Xpra client, the agent only sees the windows it needs, sparing it from your desktop clutter! GitHub: https://github.com/trycua/cua (monorepo; libs/cuabot directory) Docs: https://cua.ai/docs npm: https://www.npmjs.com/package/cuabot We'll be here to answer questions!

ed framework that standardizes environments across platforms and supports the full agent development lifecycle—benchmark, train, deploy. With Cua-Bench, you can: - Evaluate agents across multiple benchmarks with one CLI (native tasks + OSWorld + Windows Agent Arena adapters) - Test the same agent on different OS variations (Windows 11/XP/Vista, macOS themes, Linux, Android via QEMU) - Generate new tasks from natural language prompts - Create simulated environments for RL training (shell apps like Spotify, Slack with programmatic rewards) - Run oracle validations to verify environments before agent evaluation - Monitor agent runs in real-time with traces and screenshots All of this works on macOS, Linux, Windows, and Android, and is self-hostable. To get started: Install cua-bench: % pip install cua-bench Run a basic evaluation: % cb run dataset datasets/cua-bench-basic --agent demo Open the monitoring dashboard: % cb run watch <run_id> For parallelized evaluations across multiple workers: % cb run dataset datasets/cua-bench-basic --agent your-agent --max-parallel 8 Want to test across different OS variations? Just specify the environment: % cb run task slack_message --agent your-agent --env windows_xp % cb run task slack_message --agent your-agent --env macos_sonoma Generate new tasks from prompts: % cb task generate "book a flight on kayak.com" Validate environments with oracle implementations: % cb run dataset datasets/cua-bench-basic --oracle The simulated environments are particularly useful for RL training—they're HTML/JS apps that render across 10+ OS themes with programmatic reward verification. No need to spin up actual VMs for training loops. We're seeing teams use Cua-Bench for: - Training computer-use models on mobile and desktop environments - Generating large-scale training datasets (working with labs on millions of screenshots across OS variations) - RL fine-tuning with shell app simulators - Systematic evaluation across OS themes and browser versions - Building task registries (collaborating with Snorkel AI on task design and data curation, similar to their Terminal-Bench work) Cua-Bench is 100% open-source under the MIT license. We're actively developing it as part of Cua (https://github.com/trycua/cua), our Computer Use Agent SDK, and we'd love your feedback, bug reports, or feature ideas. GitHub: https://github.com/trycua/cua Docs: https://cua.ai/docs/cuabench Technical Report: https://cuabench.ai We'll be here to answer any technical questions and look forward to your comments!