clickclickclick

Oh yes, its been good for a while. When we created our Android-use[1] (like computer use) tool, it was the cheapest and the best option among Openai, Claude, llama etc. We have a planner phase followed by a "finder" phase where vision models are used. Following is the summary of our findings for planner and finder. Some of them are "work in progress" as they do not support tool calling (or are extremely bad at tool calling). +------------------------+------------------+------------------+ | Models | Planner | Finder | +------------------------+------------------+------------------+ | Gemini 1.5 Pro | recommended | recommended | | Gemini 1.5 Flash | can use | recommended | | Openai GPT 4o | recommended | work in progress | | Openai GPT 4o m

I built android-use[1] using LLM. It is pretty good at self healing due to the "loop", it constantly checks if the current step is actually a progress or regress and then determines next step. And the thing is nothing is explicitly coded, just a nudge in the prompts. 1. clickclickclick - A framework to let local LLMs control your android phone (https://github.com/BandarLabs/clickclickclick)

Not specifically trained on but most likely the Vision models have seen it. Vision models like Gemini flash/pro are already good at vision tasks on phones[1] - like clicking on UI elements and scrolling to find stuff etc. The planning of what steps to perform is also quite good with Pro model (slightly worse than GPT 4o in my opinion) 1. A framework to control your phone using Gemini - https://github.com/BandarLabs/clickclickclick

Training a base model just for computer use seems like an overkill as normal reasoning model like o3 for planning + a vision model like gemini-flash is good enough[1] without being trained specifically for computer use. But if you still want to try out this path, Google has made the screenQA dataset(rico) available[2] along with bounding boxes. 1. A framework to use local/hosted models for android use/control - https://github.com/BandarLabs/clickclickclick 2. https://github.com/google-research-datasets/screen_qa

Click3: open source alternative for Mobile operator Code: https://GitHub.com/BandarLabs/clickclickclick

Pre-planned steps by Planner will go wrong more often than not, as it will try to guess the UI layers from its memory/training data. Its better to just ask the "next step" by giving it current state of the UI. I have built a similar project for mobile automation [1] and the validator phase is not separate rather it's inherently baked in each step since we only ask next step based on current UI and previous actions. My Planner sometimes goes "Oh, we are still on home screen, let's find the Uber app icon". This sort of self-correcting behaviour was not programmed but the LLM does it on its own. 1. https://github.com/BandarLabs/ClickClickClick - A framework to automate mobile use via any LLM (local/remote)

VLMs are great - I have been able to use it for a similar project too [1]. And it's only going to get better. Congratulations on the product launch what's your VLM model for this? 1. A framework to use/control mobile phones via any LLM - https://github.com/BandarLabs/clickclickclick

And here I was in trenches with 2 upvotes with Show HN for https://github.com/BandarLabs/clickclickclick

> For starters, we could feed back screenshots of the generated website back to the LLM. But soon you’d want to give the LLM access to a full web browser so it can itself poke around the app, like a human would, to see what features work and which ones don’t. We've had some success[1] with the screenshot to actions - using Gemini/Molmo and ADB on phones. And human like decisions was made by GPT 4o. It also recalibrates itself and says "oh we are still at the home screen, let's find the gmail app first" 1. https://github.com/BandarLabs/clickclickclick - Letting AI control/use my phone.

If only someone could automate[1] the clicking and navigating part by writing in plaintext something like "Open airbnb and explore all the features as much as possible" :) 1. https://github.com/BandarLabs/clickclickclick - It does that and I am one of the authors.

> We represent the robot actions as text strings as shown below. An example of such a string could be a sequence of robot action token numbers: “1 128 91 241 5 101 127 217”. Training with numbers like this might be a little problematic, I have tried to fine tune GPT 4o-mini with very little success(just me?) On the other hand I found[1] Gemini and Molmo being able to locate elements on screen much better than 4o. 1. https://github.com/BandarLabs/clickclickclick

Being able to run STOA models on my mac has been so cool. I am using it to let molmo vision via `mlx-vlm` click on my phone[1]. Although its 4bit quantised, it works surprisingly well. 1. Controlling phone using local LLMs - https://github.com/BandarLabs/clickclickclick

I am working on automation of phones (open source) - https://github.com/BandarLabs/clickclickclick I haven't been able to quite get the Llama vision models working but I suppose with new releases in future, it should work as good as Gemini in finding bounding boxes of UI elements.