self-operating-computer

it's the 40th or so implementation of an old idea but it's the one that was done when the models got good enough to make it useful by someone who goes on podcasts. [1] Just like youtube was the 40th or so online video site but it's the one that was done by members of the paypal mafia and when enough people had high speed internet. and that is literally it. You can do that right now. Go through the 2023 LLM-related product announcements that didn't stick and vibe code it with 2026 models. Slap a cartoon on it, hype the shit out of it and post hard. I'd use a knockoff of "blobby the blobfish". [1] see https://github.com/simular-ai/Agent-S or https://github.com/trycua/cua or https://github.com/bytebot-ai/bytebot or https://github.com/microsoft/fara or https://github.com/e2b-dev/open-computer-use or https://github.com/777genius/os-ai-computer

Love this video > self-operating-computer This is quite different than https://github.com/OthersideAI/self-operating-computer Self-operating-computer uses pixel mapping to control your computer. This is a very good approach, but it's extremely unreliable. GPT-4V frequently hallucinates pixel outputs, causing it to miss interactions, or enter fail-loops >The approach by AI Jason AI Jason is using image-only methods to interact with the browser. This is a great first step, but this approach tends to be rife with hallucinations or errors. We do dom parsing in addition to image anaylsis to help GPT-4V correlate information in the image to the interactable elements within the DOM. This dramatically boosts its ability to perform the same task over and over again reliably (which proved impossible with the image-only approach)