SoM

VLMs so far have never been good at counting objects or spatial relationships (e.g. the coffee is to the right of the microwave). There are ways to help the VLM out - Set of Marks [0] from Microsoft being the most prominent, which uses segmentation to outline and label sections of the image before sending to the VLM. Providing "speakable" labels to regions helps ground the visual abilities of VLMs and is why in this paper the performance is so much better when words are present in the grid for "Task 6: Counting the rows and columns of a grid" 0: https://github.com/microsoft/SoM

At https://openadapt.ai/ we are using LLMs to automate repetitive tasks in GUI interfaces. Think robotic process automation, but via learning from demonstration rather than no-code scripting. The stack is mostly python running locally, and calling the OpenAI API (although we have plans to support offline models). For better visual understanding, we use a custom fork of Set-of-Mark prompting (https://github.com/microsoft/SoM) deployed to EC2 (see https://github.com/OpenAdaptAI/SoM/pull/3).

Here are other related links: code: https://github.com/microsoft/SoM demo: https://user-images.githubusercontent.com/3894247/281586831-...